[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:17:19 dist-test-slave-t8d9 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:17:19 dist-test-slave-t8d9 krb5kdc[399](info): set up 2 sockets
Sep 25 13:17:19 dist-test-slave-t8d9 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 13:17:20.798251 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:17:20 dist-test-slave-t8d9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:17:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:17:20.912227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39487
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:20.992702 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:20.992938 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:20.992961 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:20.994161 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:20.994202 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:20.994216 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:20.994227 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:20.994238 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:20.995713 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:20.995924 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:20.996129 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:21.001376 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:21.001451 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:21.001590 412 server_base.cc:1034] running on GCE node
W20260925 13:17:21.001379 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:21.001868 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:21.002164 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:21.003350 412 hybrid_clock.cc:648] HybridClock initialized: now 1790342241003326 us; error 53 us; skew 500 ppm
Sep 25 13:17:21 dist-test-slave-t8d9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:21.006238 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:17:21.006796 412 webserver.cc:460] Webserver started at http://127.0.98.62:38543/ using document root <none> and password file <none>
I20260925 13:17:21.007018 412 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:21.007068 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:21.007218 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:21.008333 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "0a8542da03844967949830ff25666bec"
format_stamp: "Formatted at 2026-09-25 13:17:21 on dist-test-slave-t8d9"
server_key: "fdf2835ea0aa0d88207077fdc720c638"
server_key_iv: "6ba7c6aaec3a6da047f8af31aa2b6705"
server_key_version: "encryptionkey@0"
I20260925 13:17:21.008684 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a8542da03844967949830ff25666bec"
format_stamp: "Formatted at 2026-09-25 13:17:21 on dist-test-slave-t8d9"
server_key: "fdf2835ea0aa0d88207077fdc720c638"
server_key_iv: "6ba7c6aaec3a6da047f8af31aa2b6705"
server_key_version: "encryptionkey@0"
I20260925 13:17:21.010146 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:17:21.010907 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:21.011096 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:17:21.011183 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "0a8542da03844967949830ff25666bec"
format_stamp: "Formatted at 2026-09-25 13:17:21 on dist-test-slave-t8d9"
server_key: "fdf2835ea0aa0d88207077fdc720c638"
server_key_iv: "6ba7c6aaec3a6da047f8af31aa2b6705"
server_key_version: "encryptionkey@0"
I20260925 13:17:21.011245 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:21.028470 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:21.030550 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:21.030700 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:21.034528 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39487
I20260925 13:17:21.034560 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39487 every 8 connection(s)
I20260925 13:17:21.034904 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:21.035805 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:21.037169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 412
I20260925 13:17:21.037250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:21.037914 392 external_mini_cluster.cc:1428] Setting key d7d8a9748a8027a20a5a5dd7ed0aec12
I20260925 13:17:21.038257 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec: Bootstrap starting.
I20260925 13:17:21.038888 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:21.039188 480 log.cc:824] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:21.039830 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec: No bootstrap required, opened a new log
I20260925 13:17:21.041571 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8542da03844967949830ff25666bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39487 } }
I20260925 13:17:21.041757 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:21.041800 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8542da03844967949830ff25666bec, State: Initialized, Role: FOLLOWER
I20260925 13:17:21.041923 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8542da03844967949830ff25666bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39487 } }
I20260925 13:17:21.041998 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:21.042021 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:21.042066 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:21.042869 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8542da03844967949830ff25666bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39487 } }
I20260925 13:17:21.042955 480 leader_election.cc:302] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a8542da03844967949830ff25666bec; no voters:
I20260925 13:17:21.043123 480 leader_election.cc:288] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:21.043159 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:21.043267 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [term 1 LEADER]: Becoming Leader. State: Replica: 0a8542da03844967949830ff25666bec, State: Running, Role: LEADER
I20260925 13:17:21.043347 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8542da03844967949830ff25666bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39487 } }
I20260925 13:17:21.043397 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:21.043674 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a8542da03844967949830ff25666bec. Latest consensus state: current_term: 1 leader_uuid: "0a8542da03844967949830ff25666bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8542da03844967949830ff25666bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39487 } } }
I20260925 13:17:21.043751 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:21.043686 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a8542da03844967949830ff25666bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a8542da03844967949830ff25666bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39487 } } }
I20260925 13:17:21.043926 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec [sys.catalog]: This master's current role is: LEADER
W20260925 13:17:21.045044 500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:17:21.045137 500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:17:21.045271 490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:21.045446 490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:21.046922 490 catalog_manager.cc:1347] Generated new cluster ID: 81ec5701486145dbad21712f41954153
I20260925 13:17:21.046974 490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:17:21 dist-test-slave-t8d9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:21.053324 490 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:21.054018 490 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:21.055008 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:21.038724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.0.98.62:39487)
0925 13:17:21.038905 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:21.038915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:21.039124 (+ 209us) server_negotiation.cc:407] Connection header received
0925 13:17:21.041505 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:21.041513 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:21.041598 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:21.041753 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:21.043405 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.044560 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.046157 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.046693 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.046795 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:21.050759 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:21.050763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:21.050764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:21.050775 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:21.052676 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:21.053144 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:21.053147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:21.053148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:21.053261 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:21.053560 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:21.053561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:21.053561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:21.053815 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:21.053963 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:21.054640 (+ 677us) server_negotiation.cc:299] Negotiation successful
0925 13:17:21.054776 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260925 13:17:21.058650 490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a8542da03844967949830ff25666bec: Generated new TSK 0
I20260925 13:17:21.058804 490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:17:21.082358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39487
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:21.166505 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:21.166720 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:21.166757 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:21.167953 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:21.167995 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:21.168031 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:17:21.169516 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39487
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:21.169802 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:21.170027 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:21.175115 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:21.175118 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:21.175139 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:21.175347 506 server_base.cc:1034] running on GCE node
I20260925 13:17:21.175555 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:21.175776 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:21.176929 506 hybrid_clock.cc:648] HybridClock initialized: now 1790342241176903 us; error 35 us; skew 500 ppm
Sep 25 13:17:21 dist-test-slave-t8d9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:21.179512 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:17:21.179975 506 webserver.cc:460] Webserver started at http://127.0.98.1:41435/ using document root <none> and password file <none>
I20260925 13:17:21.180191 506 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:21.180240 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:21.180367 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:21.181416 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "a152ef5712354ec7a05b4c4167595a28"
format_stamp: "Formatted at 2026-09-25 13:17:21 on dist-test-slave-t8d9"
server_key: "4a716cd961d106ce7268674067759d6b"
server_key_iv: "066eed45981fcee78d312f9ac9bd1f0d"
server_key_version: "encryptionkey@0"
I20260925 13:17:21.181807 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a152ef5712354ec7a05b4c4167595a28"
format_stamp: "Formatted at 2026-09-25 13:17:21 on dist-test-slave-t8d9"
server_key: "4a716cd961d106ce7268674067759d6b"
server_key_iv: "066eed45981fcee78d312f9ac9bd1f0d"
server_key_version: "encryptionkey@0"
I20260925 13:17:21.183079 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:21.183750 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:21.183907 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:21.183990 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "a152ef5712354ec7a05b4c4167595a28"
format_stamp: "Formatted at 2026-09-25 13:17:21 on dist-test-slave-t8d9"
server_key: "4a716cd961d106ce7268674067759d6b"
server_key_iv: "066eed45981fcee78d312f9ac9bd1f0d"
server_key_version: "encryptionkey@0"
I20260925 13:17:21.184051 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:21.197713 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:21.199924 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:21.200063 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:21.200280 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:21.200587 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:21.200619 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:21.200649 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:21.200666 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:21.206252 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33063
I20260925 13:17:21.206300 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33063 every 8 connection(s)
I20260925 13:17:21.206622 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:17:21.207697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 506
I20260925 13:17:21.207775 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:17:21.207913 392 external_mini_cluster.cc:1428] Setting key 605b46f34bfb2ce458424d6a4d5fb741
Sep 25 13:17:21 dist-test-slave-t8d9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 13:17:21.217033 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:21.207335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48223 (local address 127.0.98.62:39487)
0925 13:17:21.207382 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:21.207384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:21.207598 (+ 214us) server_negotiation.cc:407] Connection header received
0925 13:17:21.208149 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:21.208150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:21.208163 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:21.208213 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:21.209198 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.209751 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.211200 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.211487 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.211558 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:21.214077 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:21.214079 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:21.214080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:21.214089 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:21.215673 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:21.216100 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:21.216102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:21.216102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:21.216164 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:21.216378 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:21.216379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:21.216379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:21.216531 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:21.216604 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:21.216868 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 13:17:21.216958 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:17:21.217204 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:21.207283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39487 (local address 127.0.98.1:48223)
0925 13:17:21.207460 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:21.207505 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:21.208051 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:21.208292 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:21.208296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:21.208883 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:21.209094 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:21.209099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.209838 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.209839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:21.211116 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:21.211119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.211590 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.211591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:21.211732 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:21.212284 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:21.212296 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:21.213967 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:21.215768 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:21.215775 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:21.215777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:21.216021 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:21.216225 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:21.216226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:21.216226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:21.216327 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:21.216598 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:21.216604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:21.216740 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 13:17:21.216835 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 13:17:21.216986 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260925 13:17:21.217687 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39487
I20260925 13:17:21.217787 635 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:21.217940 635 heartbeater.cc:511] Master 127.0.98.62:39487 requested a full tablet report, sending...
I20260925 13:17:21.218432 444 ts_manager.cc:194] Registered new tserver with Master: a152ef5712354ec7a05b4c4167595a28 (127.0.98.1:33063)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 13:17:21.219094 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:17:21.232548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39487
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:21.313522 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:21.313764 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:21.313796 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:21.315011 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:21.315055 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:21.315091 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:17:21.316556 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39487
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:21.316802 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:21.317013 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:21.321743 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:21.321784 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:21.321789 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:21.321822 640 server_base.cc:1034] running on GCE node
I20260925 13:17:21.322141 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:21.322382 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:21.323540 640 hybrid_clock.cc:648] HybridClock initialized: now 1790342241323521 us; error 29 us; skew 500 ppm
Sep 25 13:17:21 dist-test-slave-t8d9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:21.326196 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:17:21.326665 640 webserver.cc:460] Webserver started at http://127.0.98.2:36959/ using document root <none> and password file <none>
I20260925 13:17:21.326898 640 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:21.326951 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:21.327066 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:21.328156 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "72fc2eb9ff4d4df28c6a1675fc08efc5"
format_stamp: "Formatted at 2026-09-25 13:17:21 on dist-test-slave-t8d9"
server_key: "0c9800063baa21e59b2f9803fab15459"
server_key_iv: "2235a57d33204abe3247039e3bfa4f4d"
server_key_version: "encryptionkey@0"
I20260925 13:17:21.328491 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "72fc2eb9ff4d4df28c6a1675fc08efc5"
format_stamp: "Formatted at 2026-09-25 13:17:21 on dist-test-slave-t8d9"
server_key: "0c9800063baa21e59b2f9803fab15459"
server_key_iv: "2235a57d33204abe3247039e3bfa4f4d"
server_key_version: "encryptionkey@0"
I20260925 13:17:21.329864 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:21.330619 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:21.330812 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:21.330895 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "72fc2eb9ff4d4df28c6a1675fc08efc5"
format_stamp: "Formatted at 2026-09-25 13:17:21 on dist-test-slave-t8d9"
server_key: "0c9800063baa21e59b2f9803fab15459"
server_key_iv: "2235a57d33204abe3247039e3bfa4f4d"
server_key_version: "encryptionkey@0"
I20260925 13:17:21.330951 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:21.345875 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:21.347820 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:21.347952 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:21.348152 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:21.348460 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:21.348495 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:21.348517 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:21.348538 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:21.355173 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37151
I20260925 13:17:21.355265 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37151 every 8 connection(s)
I20260925 13:17:21.355544 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:17:21.357316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 640
I20260925 13:17:21.357393 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:17:21.357558 392 external_mini_cluster.cc:1428] Setting key 26b22a2c11800bcfb105b229d09b7e73
Sep 25 13:17:21 dist-test-slave-t8d9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 13:17:21.365789 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:21.356303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42799 (local address 127.0.98.62:39487)
0925 13:17:21.356386 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:21.356388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:21.356511 (+ 123us) server_negotiation.cc:407] Connection header received
0925 13:17:21.356965 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:21.356966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:21.356975 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:21.357023 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:21.358118 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.358807 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.359882 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.360093 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.360151 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:21.362729 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:21.362732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:21.362732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:21.362741 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:21.364141 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:21.364685 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:21.364686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:21.364687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:21.364754 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:21.365062 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:21.365063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:21.365063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:21.365177 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:21.365224 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:21.365637 (+ 413us) server_negotiation.cc:299] Negotiation successful
0925 13:17:21.365719 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 13:17:21.366009 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:21.356235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39487 (local address 127.0.98.2:42799)
0925 13:17:21.356385 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:21.356426 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:21.356890 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:21.357087 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:21.357091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:21.357829 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:21.358050 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:21.358056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.358918 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.358919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:21.359792 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:21.359795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.360171 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.360172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:21.360304 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:21.360821 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:21.360832 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:21.362592 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:21.364242 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:21.364253 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:21.364257 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:21.364583 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:21.364829 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:21.364829 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:21.364830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:21.364996 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:21.365234 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:21.365241 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:21.365423 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 13:17:21.365579 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 13:17:21.365773 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260925 13:17:21.366438 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39487
I20260925 13:17:21.366530 769 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:21.366720 769 heartbeater.cc:511] Master 127.0.98.62:39487 requested a full tablet report, sending...
I20260925 13:17:21.367115 444 ts_manager.cc:194] Registered new tserver with Master: 72fc2eb9ff4d4df28c6a1675fc08efc5 (127.0.98.2:37151)
I20260925 13:17:21.367558 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:17:21.382436 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39487
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:21.465601 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:21.465837 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:21.465862 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:21.467115 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:21.467162 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:21.467201 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:17:21.468713 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39487
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:21.468971 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:21.469197 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:21.474157 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:21.474165 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:21.474165 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:21.474429 774 server_base.cc:1034] running on GCE node
I20260925 13:17:21.474638 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:21.474880 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:21.476038 774 hybrid_clock.cc:648] HybridClock initialized: now 1790342241476020 us; error 32 us; skew 500 ppm
Sep 25 13:17:21 dist-test-slave-t8d9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:21.478740 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:17:21.479228 774 webserver.cc:460] Webserver started at http://127.0.98.3:34453/ using document root <none> and password file <none>
I20260925 13:17:21.479465 774 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:21.479518 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:21.479651 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:21.480747 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "20f01cba03ca484cb303ff6df8ac7923"
format_stamp: "Formatted at 2026-09-25 13:17:21 on dist-test-slave-t8d9"
server_key: "554864e50ef33a8fb2af8c35cfd3e803"
server_key_iv: "bf4f00187946a2df171dfd13191ce415"
server_key_version: "encryptionkey@0"
I20260925 13:17:21.481092 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "20f01cba03ca484cb303ff6df8ac7923"
format_stamp: "Formatted at 2026-09-25 13:17:21 on dist-test-slave-t8d9"
server_key: "554864e50ef33a8fb2af8c35cfd3e803"
server_key_iv: "bf4f00187946a2df171dfd13191ce415"
server_key_version: "encryptionkey@0"
I20260925 13:17:21.482544 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:17:21.483286 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:21.483461 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:17:21.483547 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "20f01cba03ca484cb303ff6df8ac7923"
format_stamp: "Formatted at 2026-09-25 13:17:21 on dist-test-slave-t8d9"
server_key: "554864e50ef33a8fb2af8c35cfd3e803"
server_key_iv: "bf4f00187946a2df171dfd13191ce415"
server_key_version: "encryptionkey@0"
I20260925 13:17:21.483608 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:21.496906 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:21.499277 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:21.499439 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:21.499671 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:21.499977 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:21.500008 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:21.500037 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:21.500054 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:21.506731 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34085
I20260925 13:17:21.506779 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34085 every 8 connection(s)
I20260925 13:17:21.507110 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:17:21.507356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 774
I20260925 13:17:21.507455 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:17:21.507618 392 external_mini_cluster.cc:1428] Setting key 7f624ecf24d910a59885a61fe5f9c229
Sep 25 13:17:21 dist-test-slave-t8d9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:21.516574 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:21.507869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35235 (local address 127.0.98.62:39487)
0925 13:17:21.507943 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:21.507945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:21.508090 (+ 145us) server_negotiation.cc:407] Connection header received
0925 13:17:21.508654 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:21.508654 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:21.508664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:21.508700 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:21.509730 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.510236 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.511223 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.511430 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.511488 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:21.513640 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:21.513643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:21.513643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:21.513651 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:21.514962 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:21.515440 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:21.515441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:21.515442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:21.515513 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:21.515893 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:21.515894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:21.515894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:21.516027 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:21.516091 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:21.516424 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 13:17:21.516502 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 13:17:21.516754 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:21.507822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39487 (local address 127.0.98.3:35235)
0925 13:17:21.507977 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:21.508020 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:21.508563 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:21.508781 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:21.508786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:21.509371 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:21.509641 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:21.509647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.510308 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.510308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:21.511147 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:21.511149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.511520 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.511521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:21.511639 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:21.512116 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:21.512124 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:21.513517 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:21.515044 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:21.515053 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:21.515056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:21.515347 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:21.515607 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:21.515608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:21.515609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:21.515785 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:21.516099 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:21.516106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:21.516258 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 13:17:21.516373 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 13:17:21.516525 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260925 13:17:21.517235 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39487
I20260925 13:17:21.517380 903 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:21.517602 903 heartbeater.cc:511] Master 127.0.98.62:39487 requested a full tablet report, sending...
I20260925 13:17:21.518018 444 ts_manager.cc:194] Registered new tserver with Master: 20f01cba03ca484cb303ff6df8ac7923 (127.0.98.3:34085)
I20260925 13:17:21.518445 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35235
I20260925 13:17:21.519608 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 13:17:21 dist-test-slave-t8d9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 13:17:21 dist-test-slave-t8d9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:17:21.543730 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:21.535094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.0.98.1:33063)
0925 13:17:21.535275 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:21.535279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:21.535293 (+ 14us) server_negotiation.cc:407] Connection header received
0925 13:17:21.535368 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:21.535370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:21.535386 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:21.535448 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:21.536042 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.536787 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.537647 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.538004 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.538060 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:21.540636 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:21.540648 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:21.540648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:21.540657 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:21.542325 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:21.542762 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:21.542767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:21.542767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:21.542835 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:21.543132 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:21.543134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:21.543134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:21.543308 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:21.543488 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:21.543536 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 13:17:21.543618 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
W20260925 13:17:21.544199 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52112
Sep 25 13:17:21 dist-test-slave-t8d9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 13:17:21 dist-test-slave-t8d9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:17:21.567003 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:21.560162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52120 (local address 127.0.98.1:33063)
0925 13:17:21.560233 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:21.560235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:21.560329 (+ 94us) server_negotiation.cc:407] Connection header received
0925 13:17:21.560364 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:21.560365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:21.560378 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:21.560442 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:21.560928 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.561407 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.562251 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:21.562467 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:21.562525 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:21.564890 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:21.564891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:21.564892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:21.564898 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:21.566158 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:21.566456 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:21.566457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:21.566457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:21.566500 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:21.566627 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:21.566628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:21.566628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:21.566712 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:21.566791 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:21.566870 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 13:17:21.566931 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 13:17:21.567548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 506
I20260925 13:17:21.571813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 640
I20260925 13:17:21.576606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 774
I20260925 13:17:21.580713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 412
2026-09-25T13:17:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1751 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:17:21 dist-test-slave-t8d9 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:17:21 dist-test-slave-t8d9 krb5kdc[917](info): set up 2 sockets
Sep 25 13:17:21 dist-test-slave-t8d9 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260925 13:17:22.532894 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:17:22 dist-test-slave-t8d9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:17:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:17:22.646282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34749
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:22.726822 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:22.727056 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:22.727088 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:22.728279 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:22.728323 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:22.728343 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:22.728359 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:22.728374 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:22.730003 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:22.730258 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:22.730479 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:22.735257 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:22.735256 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:22.735270 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:22.735309 930 server_base.cc:1034] running on GCE node
I20260925 13:17:22.735627 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:22.735898 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:22.737046 930 hybrid_clock.cc:648] HybridClock initialized: now 1790342242737023 us; error 35 us; skew 500 ppm
Sep 25 13:17:22 dist-test-slave-t8d9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:22.739620 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:17:22.740096 930 webserver.cc:460] Webserver started at http://127.0.98.62:45669/ using document root <none> and password file <none>
I20260925 13:17:22.740301 930 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:22.740351 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:22.740480 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:22.741588 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "81d24bb76f1d4427853e0c7424b729b5"
format_stamp: "Formatted at 2026-09-25 13:17:22 on dist-test-slave-t8d9"
server_key: "697322c0f8d6d7b677bcf7e71029f965"
server_key_iv: "2b030cd11f92a1b4d9542694cf88016e"
server_key_version: "encryptionkey@0"
I20260925 13:17:22.741947 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "81d24bb76f1d4427853e0c7424b729b5"
format_stamp: "Formatted at 2026-09-25 13:17:22 on dist-test-slave-t8d9"
server_key: "697322c0f8d6d7b677bcf7e71029f965"
server_key_iv: "2b030cd11f92a1b4d9542694cf88016e"
server_key_version: "encryptionkey@0"
I20260925 13:17:22.743216 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:17:22.743911 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:22.744066 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:17:22.744148 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "81d24bb76f1d4427853e0c7424b729b5"
format_stamp: "Formatted at 2026-09-25 13:17:22 on dist-test-slave-t8d9"
server_key: "697322c0f8d6d7b677bcf7e71029f965"
server_key_iv: "2b030cd11f92a1b4d9542694cf88016e"
server_key_version: "encryptionkey@0"
I20260925 13:17:22.744216 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:22.767128 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:22.769397 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:22.769567 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:22.773258 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34749
I20260925 13:17:22.773295 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34749 every 8 connection(s)
I20260925 13:17:22.773656 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:22.774577 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:22.776839 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5: Bootstrap starting.
I20260925 13:17:22.777402 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:22.777696 998 log.cc:824] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:22.778411 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5: No bootstrap required, opened a new log
I20260925 13:17:22.779954 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d24bb76f1d4427853e0c7424b729b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34749 } }
I20260925 13:17:22.780095 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:22.780117 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d24bb76f1d4427853e0c7424b729b5, State: Initialized, Role: FOLLOWER
I20260925 13:17:22.780248 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d24bb76f1d4427853e0c7424b729b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34749 } }
I20260925 13:17:22.780330 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:22.780354 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:22.780385 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:22.780936 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d24bb76f1d4427853e0c7424b729b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34749 } }
I20260925 13:17:22.780957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 930
I20260925 13:17:22.781028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:22.781024 998 leader_election.cc:302] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81d24bb76f1d4427853e0c7424b729b5; no voters:
I20260925 13:17:22.781168 392 external_mini_cluster.cc:1428] Setting key 435908ead2fcfd9c5d96ddcd3a03d34f
I20260925 13:17:22.781196 998 leader_election.cc:288] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:22.781239 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:22.781607 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [term 1 LEADER]: Becoming Leader. State: Replica: 81d24bb76f1d4427853e0c7424b729b5, State: Running, Role: LEADER
I20260925 13:17:22.781710 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d24bb76f1d4427853e0c7424b729b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34749 } }
I20260925 13:17:22.781847 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:22.782068 1001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81d24bb76f1d4427853e0c7424b729b5. Latest consensus state: current_term: 1 leader_uuid: "81d24bb76f1d4427853e0c7424b729b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d24bb76f1d4427853e0c7424b729b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34749 } } }
I20260925 13:17:22.782137 1001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:22.782109 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81d24bb76f1d4427853e0c7424b729b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d24bb76f1d4427853e0c7424b729b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34749 } } }
I20260925 13:17:22.782706 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:22.782872 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:22.783424 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:22.785068 1010 catalog_manager.cc:1347] Generated new cluster ID: d58c408ec9b342a7bd570c5078c1e54a
I20260925 13:17:22.785125 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:17:22 dist-test-slave-t8d9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:22.791680 1005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:22.781718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.0.98.62:34749)
0925 13:17:22.781914 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:22.781926 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:22.782330 (+ 404us) server_negotiation.cc:407] Connection header received
0925 13:17:22.782569 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:22.782578 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:22.782639 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:22.782754 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:22.783681 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:22.784885 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:22.785851 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:22.786317 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:22.786413 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:22.788592 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:22.788597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:22.788597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:22.788611 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:22.790213 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:22.790541 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:22.790544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:22.790545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:22.790639 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:22.790814 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:22.790815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:22.790815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:22.791077 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:22.791228 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:22.791326 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 13:17:22.791467 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260925 13:17:22.797065 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:22.797619 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:22.804890 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81d24bb76f1d4427853e0c7424b729b5: Generated new TSK 0
I20260925 13:17:22.805079 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:17:22.840579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34749
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:22.937356 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:22.937588 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:22.937613 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:22.938784 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:22.938827 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:22.938855 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:17:22.940255 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34749
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:22.940482 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:22.940693 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:22.945518 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:22.945585 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:22.945570 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:22.945696 1024 server_base.cc:1034] running on GCE node
I20260925 13:17:22.945921 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:22.946141 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:22.947266 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790342242947249 us; error 33 us; skew 500 ppm
Sep 25 13:17:22 dist-test-slave-t8d9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:22.949638 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:17:22.950017 1024 webserver.cc:460] Webserver started at http://127.0.98.1:46755/ using document root <none> and password file <none>
I20260925 13:17:22.950186 1024 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:22.950223 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:22.950325 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:22.951294 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce"
format_stamp: "Formatted at 2026-09-25 13:17:22 on dist-test-slave-t8d9"
server_key: "22b7b13f63269023fc5717fd4fd3a2aa"
server_key_iv: "714d5aa75bc58324c682157692bb5ea5"
server_key_version: "encryptionkey@0"
I20260925 13:17:22.951619 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce"
format_stamp: "Formatted at 2026-09-25 13:17:22 on dist-test-slave-t8d9"
server_key: "22b7b13f63269023fc5717fd4fd3a2aa"
server_key_iv: "714d5aa75bc58324c682157692bb5ea5"
server_key_version: "encryptionkey@0"
I20260925 13:17:22.952764 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:22.953347 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:22.953555 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:22.953636 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce"
format_stamp: "Formatted at 2026-09-25 13:17:22 on dist-test-slave-t8d9"
server_key: "22b7b13f63269023fc5717fd4fd3a2aa"
server_key_iv: "714d5aa75bc58324c682157692bb5ea5"
server_key_version: "encryptionkey@0"
I20260925 13:17:22.953688 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:22.977146 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:22.979210 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:22.979331 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:22.979544 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:22.979872 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:22.979907 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:22.979930 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:22.979943 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:22.985383 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39691
I20260925 13:17:22.985467 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39691 every 8 connection(s)
I20260925 13:17:22.985780 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:17:22.986027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 1024
I20260925 13:17:22.986114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:17:22.986264 392 external_mini_cluster.cc:1428] Setting key 089d9b15490cba09d67d3dd765f98880
Sep 25 13:17:22 dist-test-slave-t8d9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 13:17:22.995103 1005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:22.986828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49789 (local address 127.0.98.62:34749)
0925 13:17:22.986873 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:22.986875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:22.986884 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:22.987362 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:22.987363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:22.987374 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:22.987416 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:22.988575 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:22.989069 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:22.990177 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:22.990398 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:22.990457 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:22.992516 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:22.992517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:22.992517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:22.992528 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:22.993841 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:22.994219 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:22.994221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:22.994221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:22.994279 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:22.994536 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:22.994537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:22.994538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:22.994644 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:22.994696 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:22.994943 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 13:17:22.995036 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:17:22.995237 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:22.986583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34749 (local address 127.0.98.1:49789)
0925 13:17:22.986748 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:22.986795 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:22.987276 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:22.987493 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:22.987498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:22.988147 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:22.988472 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:22.988479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:22.989219 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:22.989221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:22.990088 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:22.990090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:22.990472 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:22.990472 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:22.990596 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:22.991090 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:22.991099 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:22.992428 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:22.993920 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:22.993927 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:22.993929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:22.994132 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:22.994344 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:22.994345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:22.994345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:22.994468 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:22.994698 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:22.994702 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:22.994826 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 13:17:22.994929 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 13:17:22.995053 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 13:17:22.995725 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34749
I20260925 13:17:22.995818 1153 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:22.996003 1153 heartbeater.cc:511] Master 127.0.98.62:34749 requested a full tablet report, sending...
I20260925 13:17:22.996521 962 ts_manager.cc:194] Registered new tserver with Master: 5a3a6438eb2247e7ab5eb9fe786e11ce (127.0.98.1:39691)
I20260925 13:17:22.997393 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:17:23.010171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34749
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:23.089574 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:23.089798 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:23.089824 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:23.091040 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:23.091086 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:23.091114 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:17:23.092590 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34749
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:23.092821 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:23.093031 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:23.097838 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:23.097864 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:23.097856 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:23.098081 1158 server_base.cc:1034] running on GCE node
I20260925 13:17:23.098280 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:23.098532 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:23.099697 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790342243099684 us; error 29 us; skew 500 ppm
Sep 25 13:17:23 dist-test-slave-t8d9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:23.102473 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:17:23.102898 1158 webserver.cc:460] Webserver started at http://127.0.98.2:33985/ using document root <none> and password file <none>
I20260925 13:17:23.103130 1158 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:23.103174 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:23.103295 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:23.104403 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "e34b3e2ad57a469fab27dd70c28146a7"
format_stamp: "Formatted at 2026-09-25 13:17:23 on dist-test-slave-t8d9"
server_key: "aa4edeb93cc8ba9b646e4323b78d479e"
server_key_iv: "1937e2e5164b091184ec42dcbbd2b8f9"
server_key_version: "encryptionkey@0"
I20260925 13:17:23.104734 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e34b3e2ad57a469fab27dd70c28146a7"
format_stamp: "Formatted at 2026-09-25 13:17:23 on dist-test-slave-t8d9"
server_key: "aa4edeb93cc8ba9b646e4323b78d479e"
server_key_iv: "1937e2e5164b091184ec42dcbbd2b8f9"
server_key_version: "encryptionkey@0"
I20260925 13:17:23.106119 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:17:23.106809 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:23.106964 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:17:23.107041 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "e34b3e2ad57a469fab27dd70c28146a7"
format_stamp: "Formatted at 2026-09-25 13:17:23 on dist-test-slave-t8d9"
server_key: "aa4edeb93cc8ba9b646e4323b78d479e"
server_key_iv: "1937e2e5164b091184ec42dcbbd2b8f9"
server_key_version: "encryptionkey@0"
I20260925 13:17:23.107115 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:23.126581 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:23.128408 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:23.128533 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:23.128753 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:23.129065 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:23.129105 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:23.129133 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:23.129151 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:23.134689 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37755
I20260925 13:17:23.134737 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37755 every 8 connection(s)
I20260925 13:17:23.135036 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:17:23 dist-test-slave-t8d9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:23.143914 1005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.135751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34853 (local address 127.0.98.62:34749)
0925 13:17:23.135822 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:23.135824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:23.136050 (+ 226us) server_negotiation.cc:407] Connection header received
0925 13:17:23.136611 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:23.136612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:23.136622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:23.136682 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:23.137684 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.138179 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.139194 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.139371 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.139417 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.141588 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:23.141590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:23.141590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:23.141597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:23.142862 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:23.143211 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:23.143212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:23.143213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:23.143257 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:23.143425 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:23.143425 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:23.143425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:23.143519 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:23.143561 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:23.143779 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 13:17:23.143839 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 13:17:23.144083 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.135719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34749 (local address 127.0.98.2:34853)
0925 13:17:23.135912 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:23.135964 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:23.136514 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:23.136750 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:23.136754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:23.137367 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:23.137595 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:23.137601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.138261 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.138262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:23.139119 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:23.139122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.139453 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.139454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:23.139568 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.140113 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:23.140122 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:23.141465 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:23.142935 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:23.142940 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:23.142942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:23.143156 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:23.143298 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:23.143299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:23.143299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:23.143385 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:23.143558 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:23.143561 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:23.143688 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 13:17:23.143764 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 13:17:23.143877 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":121,"threads_started":1}
I20260925 13:17:23.144534 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34749
I20260925 13:17:23.144619 1287 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:23.144800 1287 heartbeater.cc:511] Master 127.0.98.62:34749 requested a full tablet report, sending...
I20260925 13:17:23.145018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 1158
I20260925 13:17:23.145105 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:17:23.145179 962 ts_manager.cc:194] Registered new tserver with Master: e34b3e2ad57a469fab27dd70c28146a7 (127.0.98.2:37755)
I20260925 13:17:23.145255 392 external_mini_cluster.cc:1428] Setting key 8064f49316e290b14e4469099da76db4
I20260925 13:17:23.145633 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34853
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:17:23.168315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34749
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:23.248735 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:23.248970 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:23.249004 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:23.250280 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:23.250335 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:23.250376 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:17:23.251940 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34749
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:23.252206 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:23.252432 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:23.257309 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:23.257406 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:23.257316 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:23.257587 1292 server_base.cc:1034] running on GCE node
I20260925 13:17:23.257786 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:23.258021 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:23.259169 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790342243259149 us; error 35 us; skew 500 ppm
Sep 25 13:17:23 dist-test-slave-t8d9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:23.261739 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:17:23.262179 1292 webserver.cc:460] Webserver started at http://127.0.98.3:37023/ using document root <none> and password file <none>
I20260925 13:17:23.262435 1292 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:23.262483 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:23.262595 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:23.263676 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "0d2ad8994004449ab236716fd37ccf01"
format_stamp: "Formatted at 2026-09-25 13:17:23 on dist-test-slave-t8d9"
server_key: "b79fa0a3768ed6baaccb099fcb5534c2"
server_key_iv: "ad6975212750441ae6d063058939848d"
server_key_version: "encryptionkey@0"
I20260925 13:17:23.263991 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0d2ad8994004449ab236716fd37ccf01"
format_stamp: "Formatted at 2026-09-25 13:17:23 on dist-test-slave-t8d9"
server_key: "b79fa0a3768ed6baaccb099fcb5534c2"
server_key_iv: "ad6975212750441ae6d063058939848d"
server_key_version: "encryptionkey@0"
I20260925 13:17:23.265295 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:23.266145 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:23.266305 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:17:23.266382 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "0d2ad8994004449ab236716fd37ccf01"
format_stamp: "Formatted at 2026-09-25 13:17:23 on dist-test-slave-t8d9"
server_key: "b79fa0a3768ed6baaccb099fcb5534c2"
server_key_iv: "ad6975212750441ae6d063058939848d"
server_key_version: "encryptionkey@0"
I20260925 13:17:23.266465 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:23.276095 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:23.278370 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:23.278501 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:23.278730 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:23.279035 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:23.279068 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:23.279098 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:23.279124 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:23.284862 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36945
I20260925 13:17:23.284911 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36945 every 8 connection(s)
I20260925 13:17:23.285216 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:17:23 dist-test-slave-t8d9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:23.293059 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 1292
I20260925 13:17:23.293154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:17:23.293305 392 external_mini_cluster.cc:1428] Setting key 9db58a895ca4fc9086e123b5e17f1ee8
I20260925 13:17:23.294291 1005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.285920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33601 (local address 127.0.98.62:34749)
0925 13:17:23.285962 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:23.285964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:23.286153 (+ 189us) server_negotiation.cc:407] Connection header received
0925 13:17:23.286664 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:23.286665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:23.286677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:23.286727 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:23.287672 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.288161 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.289162 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.289362 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.289431 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.291532 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:23.291534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:23.291534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:23.291542 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:23.292760 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:23.293239 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:23.293241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:23.293241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:23.293308 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:23.293619 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:23.293620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:23.293620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:23.293768 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:23.293828 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:23.294143 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 13:17:23.294224 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:17:23.294507 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.285891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34749 (local address 127.0.98.3:33601)
0925 13:17:23.286036 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:23.286072 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:23.286565 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:23.286801 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:23.286805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:23.287374 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:23.287576 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:23.287581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.288246 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.288247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:23.289078 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:23.289080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.289441 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.289443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:23.289622 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.290122 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:23.290131 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:23.291440 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:23.292839 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:23.292846 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:23.292849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:23.293147 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:23.293380 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:23.293381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:23.293381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:23.293514 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:23.293828 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:23.293836 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:23.293999 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 13:17:23.294104 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 13:17:23.294272 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260925 13:17:23.294904 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34749
I20260925 13:17:23.294991 1421 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:23.295162 1421 heartbeater.cc:511] Master 127.0.98.62:34749 requested a full tablet report, sending...
I20260925 13:17:23.295486 961 ts_manager.cc:194] Registered new tserver with Master: 0d2ad8994004449ab236716fd37ccf01 (127.0.98.3:36945)
I20260925 13:17:23.295857 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33601
I20260925 13:17:23.305114 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:17:23.312704 1005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.306738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.0.98.62:34749)
0925 13:17:23.306800 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:23.306802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:23.306878 (+ 76us) server_negotiation.cc:407] Connection header received
0925 13:17:23.306929 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:23.306930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:23.306940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:23.306987 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:23.307513 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.308000 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.308736 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.308939 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.309002 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.310304 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:23.310306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:23.310306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:23.310314 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:23.311618 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:23.311980 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:23.311981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:23.311982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:23.312040 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:23.312183 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:23.312183 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:23.312184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:23.312348 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:23.312422 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:23.312565 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 13:17:23.312632 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:17:23.314975 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:17:23.316514 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:23.322811 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.319153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46342 (local address 127.0.98.2:37755)
0925 13:17:23.319308 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:23.319312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:23.319322 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:17:23.319363 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:23.319365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:23.319379 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:23.319449 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:23.319662 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.320457 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.321820 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.322430 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.322491 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.322552 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:23.322688 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 13:17:23.322729 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260925 13:17:23.322803 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.318881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37755 (local address 127.0.0.1:46342)
0925 13:17:23.319192 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:23.319234 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:23.319296 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:23.319455 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:23.319458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:23.319460 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:23.319533 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:23.319535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.320550 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.320551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:23.321728 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:23.321730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.322500 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.322501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:23.322590 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.322603 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 13:17:23.322666 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 13:17:23.322722 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":54,"threads_started":1}
I20260925 13:17:23.322973 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.319009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51616 (local address 127.0.98.3:36945)
0925 13:17:23.319535 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:23.319540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:23.319548 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:17:23.319579 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:23.319581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:23.319594 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:23.319655 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:23.319843 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.320889 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.321876 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.322508 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.322574 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.322625 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:23.322854 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 13:17:23.322895 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":88,"threads_started":1}
I20260925 13:17:23.323158 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.318881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36945 (local address 127.0.0.1:51616)
0925 13:17:23.319192 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:23.319234 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:23.319295 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:23.319727 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:23.319729 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:23.319731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:23.319774 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:23.319776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.320969 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.320970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:23.321816 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:23.321818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.322658 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.322659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:23.322723 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.322727 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:23.322771 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 13:17:23.322813 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":117,"thread_start_us":71,"threads_started":1}
I20260925 13:17:23.323168 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.318881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39691 (local address 127.0.0.1:43360)
0925 13:17:23.319367 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:23.319373 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:23.319409 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:23.319552 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:23.319553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:23.319555 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:23.319591 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:23.319592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.320936 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.320937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:23.322060 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:23.322062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.322944 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.322945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:23.323011 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.323014 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:23.323075 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 13:17:23.323109 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":251,"thread_start_us":45,"threads_started":1}
I20260925 13:17:23.323310 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.318999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.0.98.1:39691)
0925 13:17:23.319202 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:23.319206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:23.319424 (+ 218us) server_negotiation.cc:407] Connection header received
0925 13:17:23.319467 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:23.319469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:23.319485 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:23.319557 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:23.319632 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.320813 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.322140 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.322814 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.322868 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.322915 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:23.323205 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 13:17:23.323241 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":97,"threads_started":1}
I20260925 13:17:23.323968 1221 tablet_service.cc:1467] Processing CreateTablet for tablet c6d266ec98bc4f4591cbf24120d9763b (DEFAULT_TABLE table=test-table [id=2a7d9665a97c40cfbc4a017bafb4c46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:23.323992 1355 tablet_service.cc:1467] Processing CreateTablet for tablet c6d266ec98bc4f4591cbf24120d9763b (DEFAULT_TABLE table=test-table [id=2a7d9665a97c40cfbc4a017bafb4c46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:23.323969 1087 tablet_service.cc:1467] Processing CreateTablet for tablet c6d266ec98bc4f4591cbf24120d9763b (DEFAULT_TABLE table=test-table [id=2a7d9665a97c40cfbc4a017bafb4c46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:23.324313 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d266ec98bc4f4591cbf24120d9763b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:23.324314 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d266ec98bc4f4591cbf24120d9763b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:23.324314 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d266ec98bc4f4591cbf24120d9763b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:23.326947 1438 tablet_bootstrap.cc:492] T c6d266ec98bc4f4591cbf24120d9763b P e34b3e2ad57a469fab27dd70c28146a7: Bootstrap starting.
I20260925 13:17:23.327409 1439 tablet_bootstrap.cc:492] T c6d266ec98bc4f4591cbf24120d9763b P 0d2ad8994004449ab236716fd37ccf01: Bootstrap starting.
I20260925 13:17:23.327745 1438 tablet_bootstrap.cc:654] T c6d266ec98bc4f4591cbf24120d9763b P e34b3e2ad57a469fab27dd70c28146a7: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:23.328012 1439 tablet_bootstrap.cc:654] T c6d266ec98bc4f4591cbf24120d9763b P 0d2ad8994004449ab236716fd37ccf01: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:23.328308 1438 log.cc:824] T c6d266ec98bc4f4591cbf24120d9763b P e34b3e2ad57a469fab27dd70c28146a7: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:23.329017 1439 log.cc:824] T c6d266ec98bc4f4591cbf24120d9763b P 0d2ad8994004449ab236716fd37ccf01: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:23.329303 1440 tablet_bootstrap.cc:492] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce: Bootstrap starting.
I20260925 13:17:23.329789 1439 tablet_bootstrap.cc:492] T c6d266ec98bc4f4591cbf24120d9763b P 0d2ad8994004449ab236716fd37ccf01: No bootstrap required, opened a new log
I20260925 13:17:23.329908 1439 ts_tablet_manager.cc:1397] T c6d266ec98bc4f4591cbf24120d9763b P 0d2ad8994004449ab236716fd37ccf01: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 13:17:23.329931 1440 tablet_bootstrap.cc:654] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:23.330097 1438 tablet_bootstrap.cc:492] T c6d266ec98bc4f4591cbf24120d9763b P e34b3e2ad57a469fab27dd70c28146a7: No bootstrap required, opened a new log
I20260925 13:17:23.330164 1438 ts_tablet_manager.cc:1397] T c6d266ec98bc4f4591cbf24120d9763b P e34b3e2ad57a469fab27dd70c28146a7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 13:17:23.330365 1440 log.cc:824] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:23.331051 1440 tablet_bootstrap.cc:492] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce: No bootstrap required, opened a new log
I20260925 13:17:23.331110 1440 ts_tablet_manager.cc:1397] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:17:23.331817 1439 raft_consensus.cc:348] T c6d266ec98bc4f4591cbf24120d9763b P 0d2ad8994004449ab236716fd37ccf01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d2ad8994004449ab236716fd37ccf01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36945 } } peers { permanent_uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39691 } } peers { permanent_uuid: "e34b3e2ad57a469fab27dd70c28146a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } }
I20260925 13:17:23.332010 1439 raft_consensus.cc:374] T c6d266ec98bc4f4591cbf24120d9763b P 0d2ad8994004449ab236716fd37ccf01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:23.332048 1439 raft_consensus.cc:729] T c6d266ec98bc4f4591cbf24120d9763b P 0d2ad8994004449ab236716fd37ccf01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2ad8994004449ab236716fd37ccf01, State: Initialized, Role: FOLLOWER
I20260925 13:17:23.332054 1438 raft_consensus.cc:348] T c6d266ec98bc4f4591cbf24120d9763b P e34b3e2ad57a469fab27dd70c28146a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d2ad8994004449ab236716fd37ccf01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36945 } } peers { permanent_uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39691 } } peers { permanent_uuid: "e34b3e2ad57a469fab27dd70c28146a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } }
I20260925 13:17:23.332156 1439 consensus_queue.cc:260] T c6d266ec98bc4f4591cbf24120d9763b P 0d2ad8994004449ab236716fd37ccf01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d2ad8994004449ab236716fd37ccf01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36945 } } peers { permanent_uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39691 } } peers { permanent_uuid: "e34b3e2ad57a469fab27dd70c28146a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } }
I20260925 13:17:23.332192 1438 raft_consensus.cc:374] T c6d266ec98bc4f4591cbf24120d9763b P e34b3e2ad57a469fab27dd70c28146a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:23.332222 1438 raft_consensus.cc:729] T c6d266ec98bc4f4591cbf24120d9763b P e34b3e2ad57a469fab27dd70c28146a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e34b3e2ad57a469fab27dd70c28146a7, State: Initialized, Role: FOLLOWER
I20260925 13:17:23.332321 1438 consensus_queue.cc:260] T c6d266ec98bc4f4591cbf24120d9763b P e34b3e2ad57a469fab27dd70c28146a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d2ad8994004449ab236716fd37ccf01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36945 } } peers { permanent_uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39691 } } peers { permanent_uuid: "e34b3e2ad57a469fab27dd70c28146a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } }
I20260925 13:17:23.332414 1439 ts_tablet_manager.cc:1428] T c6d266ec98bc4f4591cbf24120d9763b P 0d2ad8994004449ab236716fd37ccf01: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:17:23.332572 1438 ts_tablet_manager.cc:1428] T c6d266ec98bc4f4591cbf24120d9763b P e34b3e2ad57a469fab27dd70c28146a7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 13:17:23.332720 1287 heartbeater.cc:503] Master 127.0.98.62:34749 was elected leader, sending a full tablet report...
I20260925 13:17:23.333091 1440 raft_consensus.cc:348] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d2ad8994004449ab236716fd37ccf01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36945 } } peers { permanent_uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39691 } } peers { permanent_uuid: "e34b3e2ad57a469fab27dd70c28146a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } }
I20260925 13:17:23.333253 1440 raft_consensus.cc:374] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:23.333285 1440 raft_consensus.cc:729] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3a6438eb2247e7ab5eb9fe786e11ce, State: Initialized, Role: FOLLOWER
I20260925 13:17:23.333410 1421 heartbeater.cc:503] Master 127.0.98.62:34749 was elected leader, sending a full tablet report...
I20260925 13:17:23.333388 1440 consensus_queue.cc:260] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d2ad8994004449ab236716fd37ccf01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36945 } } peers { permanent_uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39691 } } peers { permanent_uuid: "e34b3e2ad57a469fab27dd70c28146a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } }
I20260925 13:17:23.333652 1440 ts_tablet_manager.cc:1428] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 13:17:23.333673 1153 heartbeater.cc:503] Master 127.0.98.62:34749 was elected leader, sending a full tablet report...
I20260925 13:17:23.364246 1446 raft_consensus.cc:482] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:23.364420 1446 raft_consensus.cc:504] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d2ad8994004449ab236716fd37ccf01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36945 } } peers { permanent_uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39691 } } peers { permanent_uuid: "e34b3e2ad57a469fab27dd70c28146a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } }
I20260925 13:17:23.364740 1446 leader_election.cc:288] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d2ad8994004449ab236716fd37ccf01 (127.0.98.3:36945), e34b3e2ad57a469fab27dd70c28146a7 (127.0.98.2:37755)
I20260925 13:17:23.367820 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.364999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34535 (local address 127.0.98.2:37755)
0925 13:17:23.365055 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:23.365057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:23.365066 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:23.365092 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:23.365093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:23.365102 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:23.365165 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:23.365352 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.365921 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.367097 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.367502 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.367552 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.367562 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:23.367709 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 13:17:23.367750 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:17:23.367863 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.365003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33473 (local address 127.0.98.3:36945)
0925 13:17:23.365060 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:23.365062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:23.365324 (+ 262us) server_negotiation.cc:407] Connection header received
0925 13:17:23.365358 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:23.365359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:23.365372 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:23.365411 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:23.365600 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.366122 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.367173 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.367567 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.367641 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.367650 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:23.367771 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 13:17:23.367809 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:17:23.367842 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.364914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37755 (local address 127.0.98.1:34535)
0925 13:17:23.364970 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:23.364981 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:23.365022 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:23.365184 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:23.365185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:23.365187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:23.365270 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:23.365272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.365990 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.365991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:23.367021 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:23.367023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.367570 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.367571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:23.367645 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.367649 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:23.367697 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 13:17:23.367737 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":42}
I20260925 13:17:23.368218 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.364914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36945 (local address 127.0.98.1:33473)
0925 13:17:23.365256 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:23.365265 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:23.365316 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:23.365429 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:23.365430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:23.365432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:23.365495 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:23.365496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.366213 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.366214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:23.367097 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:23.367099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.367647 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.367648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:23.367707 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.367711 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:23.367760 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 13:17:23.367795 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":154,"thread_start_us":114,"threads_started":1}
I20260925 13:17:23.368270 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d266ec98bc4f4591cbf24120d9763b" candidate_uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e34b3e2ad57a469fab27dd70c28146a7" is_pre_election: true
I20260925 13:17:23.368444 1241 raft_consensus.cc:2454] T c6d266ec98bc4f4591cbf24120d9763b P e34b3e2ad57a469fab27dd70c28146a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a3a6438eb2247e7ab5eb9fe786e11ce in term 0.
I20260925 13:17:23.368572 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d266ec98bc4f4591cbf24120d9763b" candidate_uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d2ad8994004449ab236716fd37ccf01" is_pre_election: true
I20260925 13:17:23.368681 1042 leader_election.cc:302] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a3a6438eb2247e7ab5eb9fe786e11ce, e34b3e2ad57a469fab27dd70c28146a7; no voters:
I20260925 13:17:23.368712 1375 raft_consensus.cc:2454] T c6d266ec98bc4f4591cbf24120d9763b P 0d2ad8994004449ab236716fd37ccf01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a3a6438eb2247e7ab5eb9fe786e11ce in term 0.
I20260925 13:17:23.368822 1446 raft_consensus.cc:2781] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:23.368896 1446 raft_consensus.cc:482] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:23.368920 1446 raft_consensus.cc:3037] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:23.369683 1446 raft_consensus.cc:504] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d2ad8994004449ab236716fd37ccf01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36945 } } peers { permanent_uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39691 } } peers { permanent_uuid: "e34b3e2ad57a469fab27dd70c28146a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } }
I20260925 13:17:23.369866 1446 leader_election.cc:288] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [CANDIDATE]: Term 1 election: Requested vote from peers 0d2ad8994004449ab236716fd37ccf01 (127.0.98.3:36945), e34b3e2ad57a469fab27dd70c28146a7 (127.0.98.2:37755)
I20260925 13:17:23.369983 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d266ec98bc4f4591cbf24120d9763b" candidate_uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d2ad8994004449ab236716fd37ccf01"
I20260925 13:17:23.370030 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d266ec98bc4f4591cbf24120d9763b" candidate_uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e34b3e2ad57a469fab27dd70c28146a7"
I20260925 13:17:23.370054 1375 raft_consensus.cc:3037] T c6d266ec98bc4f4591cbf24120d9763b P 0d2ad8994004449ab236716fd37ccf01 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:23.370119 1241 raft_consensus.cc:3037] T c6d266ec98bc4f4591cbf24120d9763b P e34b3e2ad57a469fab27dd70c28146a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:23.370762 1375 raft_consensus.cc:2454] T c6d266ec98bc4f4591cbf24120d9763b P 0d2ad8994004449ab236716fd37ccf01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a3a6438eb2247e7ab5eb9fe786e11ce in term 1.
I20260925 13:17:23.370958 1040 leader_election.cc:302] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d2ad8994004449ab236716fd37ccf01, 5a3a6438eb2247e7ab5eb9fe786e11ce; no voters:
I20260925 13:17:23.371032 1446 raft_consensus.cc:2781] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:23.371172 1446 raft_consensus.cc:686] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [term 1 LEADER]: Becoming Leader. State: Replica: 5a3a6438eb2247e7ab5eb9fe786e11ce, State: Running, Role: LEADER
I20260925 13:17:23.371263 1446 consensus_queue.cc:237] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d2ad8994004449ab236716fd37ccf01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36945 } } peers { permanent_uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39691 } } peers { permanent_uuid: "e34b3e2ad57a469fab27dd70c28146a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } }
I20260925 13:17:23.371405 1241 raft_consensus.cc:2454] T c6d266ec98bc4f4591cbf24120d9763b P e34b3e2ad57a469fab27dd70c28146a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a3a6438eb2247e7ab5eb9fe786e11ce in term 1.
I20260925 13:17:23.372159 962 catalog_manager.cc:5495] T c6d266ec98bc4f4591cbf24120d9763b P 5a3a6438eb2247e7ab5eb9fe786e11ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a3a6438eb2247e7ab5eb9fe786e11ce (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d2ad8994004449ab236716fd37ccf01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a3a6438eb2247e7ab5eb9fe786e11ce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e34b3e2ad57a469fab27dd70c28146a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } health_report { overall_health: UNKNOWN } } }
Sep 25 13:17:23 dist-test-slave-t8d9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
W20260925 13:17:23.385583 1288 tablet.cc:2369] T c6d266ec98bc4f4591cbf24120d9763b P e34b3e2ad57a469fab27dd70c28146a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:17:23.386941 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.379577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.0.98.1:39691)
0925 13:17:23.379651 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:23.379653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:23.379717 (+ 64us) server_negotiation.cc:407] Connection header received
0925 13:17:23.379765 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:23.379766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:23.379776 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:23.379822 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:23.380316 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.380796 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.381524 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.381751 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.381811 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.384114 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:23.384125 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:23.384126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:23.384134 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:23.385811 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:23.386149 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:23.386152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:23.386152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:23.386213 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:23.386413 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:23.386414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:23.386414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:23.386584 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:23.386703 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:23.386806 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 13:17:23.386875 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 13:17:23 dist-test-slave-t8d9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342243, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 13:17:23 dist-test-slave-t8d9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342243, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:17:23.408154 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.401456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43364 (local address 127.0.98.1:39691)
0925 13:17:23.401587 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:23.401589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:23.401630 (+ 41us) server_negotiation.cc:407] Connection header received
0925 13:17:23.401684 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:23.401685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:23.401694 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:23.401743 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:23.402243 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.402743 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.403427 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.403631 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.403676 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.405915 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:23.405916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:23.405917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:23.405923 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:23.407216 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:23.407570 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:23.407571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:23.407571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:23.407618 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:23.407747 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:23.407747 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:23.407747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:23.407832 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:23.407900 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:23.408027 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 13:17:23.408084 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
Sep 25 13:17:23 dist-test-slave-t8d9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 13:17:23 dist-test-slave-t8d9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:17:23.467254 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:23.459763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.0.98.1:39691)
0925 13:17:23.459811 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:23.459813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:23.459936 (+ 123us) server_negotiation.cc:407] Connection header received
0925 13:17:23.460017 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:23.460018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:23.460028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:23.460074 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:23.460623 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.461129 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.461911 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:23.462094 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:23.462146 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:23.464283 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:23.464285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:23.464285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:23.464292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:23.465520 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:23.466577 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:23.466578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:23.466578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:23.466633 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:23.466791 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:23.466792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:23.466792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:23.466884 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:23.466982 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:23.467077 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 13:17:23.467150 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 13:17:23.467737 1067 tablet_service.cc:2908] Rejecting scan request for tablet c6d266ec98bc4f4591cbf24120d9763b: Uninitialized: clean time has not yet been initialized
I20260925 13:17:23.468451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 1024
I20260925 13:17:23.472811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 1158
I20260925 13:17:23.476528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 1292
I20260925 13:17:23.480422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 930
2026-09-25T13:17:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1899 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:17:25 dist-test-slave-t8d9 krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:17:25 dist-test-slave-t8d9 krb5kdc[1462](info): set up 2 sockets
Sep 25 13:17:25 dist-test-slave-t8d9 krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260925 13:17:26.074026 392 mini_kdc.cc:121] Time spent starting KDC: real 2.574s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 13:17:26.185361 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.111s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 13:17:26.295825 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.110s user 0.001s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/test-user.keytab.
W20260925 13:17:26.831218 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.458s user 0.000s sys 0.001s
Sep 25 13:17:26 dist-test-slave-t8d9 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:17:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260925 13:17:27.556317 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.706s user 0.000s sys 0.003s
I20260925 13:17:27.556682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33549
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:27.640868 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:27.641077 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:27.641098 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:27.642297 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:27.642345 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 13:17:27.642359 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:27.642375 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:27.642391 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:27.642405 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:27.644133 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33549
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:27.644374 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:27.644593 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:27.649363 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:27.649392 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:27.649389 1475 server_base.cc:1034] running on GCE node
W20260925 13:17:27.649363 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:27.649796 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:27.650080 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:27.651232 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790342247651197 us; error 52 us; skew 500 ppm
Sep 25 13:17:27 dist-test-slave-t8d9 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:27.658901 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:17:27.659358 1475 webserver.cc:460] Webserver started at http://127.0.98.62:40159/ using document root <none> and password file <none>
I20260925 13:17:27.659557 1475 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:27.659615 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:27.659739 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:27.660806 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "03112d80310c497792cc5b871ae0ccc4"
format_stamp: "Formatted at 2026-09-25 13:17:27 on dist-test-slave-t8d9"
server_key: "fd9e15e323252373583db2a9a89c0b63"
server_key_iv: "f700f7432fc739b13c143de213881423"
server_key_version: "encryptionkey@0"
I20260925 13:17:27.661137 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "03112d80310c497792cc5b871ae0ccc4"
format_stamp: "Formatted at 2026-09-25 13:17:27 on dist-test-slave-t8d9"
server_key: "fd9e15e323252373583db2a9a89c0b63"
server_key_iv: "f700f7432fc739b13c143de213881423"
server_key_version: "encryptionkey@0"
I20260925 13:17:27.662483 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:27.663237 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:27.663415 1475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:27.663498 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "03112d80310c497792cc5b871ae0ccc4"
format_stamp: "Formatted at 2026-09-25 13:17:27 on dist-test-slave-t8d9"
server_key: "fd9e15e323252373583db2a9a89c0b63"
server_key_iv: "f700f7432fc739b13c143de213881423"
server_key_version: "encryptionkey@0"
I20260925 13:17:27.663559 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:27.673257 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:27.675072 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:27.675220 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:27.679975 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33549
I20260925 13:17:27.680045 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33549 every 8 connection(s)
I20260925 13:17:27.680344 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:27.680964 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:27.681201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 1475
I20260925 13:17:27.681277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:27.681419 392 external_mini_cluster.cc:1428] Setting key d7b43fc9090f09597217988382b62149
I20260925 13:17:27.683806 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4: Bootstrap starting.
I20260925 13:17:27.684401 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:27.684726 1553 log.cc:824] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:27.685451 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4: No bootstrap required, opened a new log
Sep 25 13:17:27 dist-test-slave-t8d9 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:27.687785 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03112d80310c497792cc5b871ae0ccc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } }
I20260925 13:17:27.687932 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:27.687973 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03112d80310c497792cc5b871ae0ccc4, State: Initialized, Role: FOLLOWER
I20260925 13:17:27.688112 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03112d80310c497792cc5b871ae0ccc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } }
I20260925 13:17:27.688195 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:27.688233 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:27.688282 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:27.690999 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:27.681949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.0.98.62:33549)
0925 13:17:27.682123 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:27.682131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:27.682156 (+ 25us) server_negotiation.cc:407] Connection header received
0925 13:17:27.682566 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:27.682572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:27.682622 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:27.682739 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:27.683476 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:27.684375 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:27.685289 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:27.685735 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:27.685815 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:27.688418 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:27.688424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:27.688424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:27.688440 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:27.689874 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:27.690179 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:27.690181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:27.690181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:27.690256 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:27.690374 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:27.690374 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:27.690375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:27.690583 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:27.690680 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:27.690752 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 13:17:27.690847 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":57,"threads_started":1}
I20260925 13:17:27.726224 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03112d80310c497792cc5b871ae0ccc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } }
I20260925 13:17:27.726418 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03112d80310c497792cc5b871ae0ccc4; no voters:
I20260925 13:17:27.726624 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:27.726809 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:27.726971 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:27.726977 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [term 1 LEADER]: Becoming Leader. State: Replica: 03112d80310c497792cc5b871ae0ccc4, State: Running, Role: LEADER
I20260925 13:17:27.727180 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03112d80310c497792cc5b871ae0ccc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } }
I20260925 13:17:27.727644 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03112d80310c497792cc5b871ae0ccc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03112d80310c497792cc5b871ae0ccc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } } }
I20260925 13:17:27.727726 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:27.727873 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03112d80310c497792cc5b871ae0ccc4. Latest consensus state: current_term: 1 leader_uuid: "03112d80310c497792cc5b871ae0ccc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03112d80310c497792cc5b871ae0ccc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } } }
I20260925 13:17:27.727931 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:27.728049 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:27.728576 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:27.730489 1567 catalog_manager.cc:1347] Generated new cluster ID: a533d4f32e934b228a7a78022d564e55
I20260925 13:17:27.730545 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:17:27.764703 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:27.765456 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:27.778513 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4: Generated new TSK 0
I20260925 13:17:27.778746 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260925 13:17:28.356752 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.571s user 0.000s sys 0.002s
I20260925 13:17:28.357076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33549
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:28.438254 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:28.438475 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:28.438501 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:28.439705 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:28.439747 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:28.439767 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:17:28.439785 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:17:28.441311 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:28.441619 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:28.441848 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:28.446718 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:28.446856 1579 server_base.cc:1034] running on GCE node
W20260925 13:17:28.446703 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:28.446704 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:28.447196 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:28.447435 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:28.448609 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790342248448597 us; error 32 us; skew 500 ppm
Sep 25 13:17:28 dist-test-slave-t8d9 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:28.451222 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:17:28.451767 1579 webserver.cc:460] Webserver started at http://127.0.98.1:38923/ using document root <none> and password file <none>
I20260925 13:17:28.451980 1579 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:28.452021 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:28.452126 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:28.453174 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "8c1bb58fa8f947f1b2ec7718f60a5600"
format_stamp: "Formatted at 2026-09-25 13:17:28 on dist-test-slave-t8d9"
server_key: "cbb54b15e28eb2a6d3ae55ce62cca777"
server_key_iv: "e8e88ab3d0d923865d9831dc13daf879"
server_key_version: "encryptionkey@0"
I20260925 13:17:28.453471 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8c1bb58fa8f947f1b2ec7718f60a5600"
format_stamp: "Formatted at 2026-09-25 13:17:28 on dist-test-slave-t8d9"
server_key: "cbb54b15e28eb2a6d3ae55ce62cca777"
server_key_iv: "e8e88ab3d0d923865d9831dc13daf879"
server_key_version: "encryptionkey@0"
I20260925 13:17:28.454815 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:17:28.455442 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:28.455606 1579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:17:28.455685 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "8c1bb58fa8f947f1b2ec7718f60a5600"
format_stamp: "Formatted at 2026-09-25 13:17:28 on dist-test-slave-t8d9"
server_key: "cbb54b15e28eb2a6d3ae55ce62cca777"
server_key_iv: "e8e88ab3d0d923865d9831dc13daf879"
server_key_version: "encryptionkey@0"
I20260925 13:17:28.455750 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:28.476264 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:28.478178 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:28.478307 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:28.478901 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:28.478950 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:28.478979 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:28.479013 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 13:17:28 dist-test-slave-t8d9 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:28.485904 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38825
I20260925 13:17:28.485958 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38825 every 8 connection(s)
I20260925 13:17:28.486320 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:17:28.488128 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:28.479292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56159 (local address 127.0.98.62:33549)
0925 13:17:28.479430 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:28.479433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:28.479605 (+ 172us) server_negotiation.cc:407] Connection header received
0925 13:17:28.480266 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:28.480267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:28.480277 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:28.480347 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:28.481225 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:28.481923 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:28.482936 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:28.483143 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:28.483206 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:28.485507 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:28.485508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:28.485509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:28.485516 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:28.487080 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:28.487409 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:28.487409 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:28.487410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:28.487455 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:28.487646 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:28.487646 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:28.487647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:28.487745 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:28.487790 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:28.487990 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 13:17:28.488048 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260925 13:17:28.488267 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:28.479284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33549 (local address 127.0.98.1:56159)
0925 13:17:28.479453 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:28.479503 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:28.480168 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:28.480440 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:28.480444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:28.480792 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:28.481121 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:28.481129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:28.482010 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:28.482011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:28.482847 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:28.482850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:28.483230 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:28.483231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:28.483346 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:28.483817 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:28.483828 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:28.485417 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:28.487149 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:28.487152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:28.487154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:28.487354 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:28.487491 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:28.487491 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:28.487491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:28.487605 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:28.487791 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:28.487794 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:28.487903 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 13:17:28.487972 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 13:17:28.488088 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260925 13:17:28.488709 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33549
I20260925 13:17:28.488792 1711 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:28.488956 1711 heartbeater.cc:511] Master 127.0.98.62:33549 requested a full tablet report, sending...
I20260925 13:17:28.489460 1506 ts_manager.cc:194] Registered new tserver with Master: 8c1bb58fa8f947f1b2ec7718f60a5600 (127.0.98.1:38825)
I20260925 13:17:28.490259 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56159
I20260925 13:17:28.491729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 1579
I20260925 13:17:28.491804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:17:28.491952 392 external_mini_cluster.cc:1428] Setting key e19f613fc8a4988cf9847fe448e68d5d
I20260925 13:17:28.495361 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:28.489720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33559 (local address 127.0.98.62:33549)
0925 13:17:28.489760 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:28.489762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:28.489826 (+ 64us) server_negotiation.cc:407] Connection header received
0925 13:17:28.489870 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:28.489871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:28.489880 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:28.489941 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:28.490190 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:28.490783 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:28.491794 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:28.492040 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:28.492103 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:28.493108 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:28.493109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:28.493110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:28.493116 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:28.494389 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:28.494692 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:28.494693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:28.494693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:28.494745 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:28.494897 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:28.494898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:28.494898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:28.495004 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:28.495057 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:28.495190 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 13:17:28.495267 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:17:28.495375 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:28.489655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33549 (local address 127.0.98.1:33559)
0925 13:17:28.489776 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:28.489786 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:28.489835 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:28.489945 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:28.489945 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:28.490091 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:28.490147 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:28.490149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:28.490873 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:28.490874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:28.491438 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:28.491440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:28.492190 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:28.492190 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:28.492257 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:28.492731 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:28.492735 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:28.493025 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:28.494464 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:28.494466 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:28.494466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:28.494634 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:28.494795 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:28.494796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:28.494796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:28.494851 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:28.495060 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:28.495061 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:28.495105 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 13:17:28.495163 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 13:17:28.495268 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260925 13:17:29.217885 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.725s user 0.000s sys 0.002s
I20260925 13:17:29.218219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33549
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:29.297348 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:29.297600 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:29.297626 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:29.298816 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:29.298864 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:29.298884 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:17:29.298902 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:17:29.300441 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:29.300673 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:29.300900 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:29.305790 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:29.305824 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:29.305823 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:29.305855 1720 server_base.cc:1034] running on GCE node
I20260925 13:17:29.306183 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:29.306424 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:29.307596 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790342249307582 us; error 33 us; skew 500 ppm
Sep 25 13:17:29 dist-test-slave-t8d9 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:29.310078 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:17:29.310524 1720 webserver.cc:460] Webserver started at http://127.0.98.2:39363/ using document root <none> and password file <none>
I20260925 13:17:29.310724 1720 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:29.310763 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:29.310858 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:29.311882 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "a66ecc6d04e74e739b72776e786ad234"
format_stamp: "Formatted at 2026-09-25 13:17:29 on dist-test-slave-t8d9"
server_key: "34cea991fdd987d3d01b724b61793890"
server_key_iv: "55f6ed21d22bea203ba3ab1347dc3e28"
server_key_version: "encryptionkey@0"
I20260925 13:17:29.312187 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a66ecc6d04e74e739b72776e786ad234"
format_stamp: "Formatted at 2026-09-25 13:17:29 on dist-test-slave-t8d9"
server_key: "34cea991fdd987d3d01b724b61793890"
server_key_iv: "55f6ed21d22bea203ba3ab1347dc3e28"
server_key_version: "encryptionkey@0"
I20260925 13:17:29.313422 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:17:29.314205 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:29.314364 1720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:17:29.314440 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "a66ecc6d04e74e739b72776e786ad234"
format_stamp: "Formatted at 2026-09-25 13:17:29 on dist-test-slave-t8d9"
server_key: "34cea991fdd987d3d01b724b61793890"
server_key_iv: "55f6ed21d22bea203ba3ab1347dc3e28"
server_key_version: "encryptionkey@0"
I20260925 13:17:29.314507 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:29.336562 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:29.338634 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:29.338776 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:29.339388 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:29.339447 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:29.339479 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:29.339496 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:29.346369 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43455
I20260925 13:17:29.346449 1851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43455 every 8 connection(s)
Sep 25 13:17:29 dist-test-slave-t8d9 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:29.346753 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:17:29.349663 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.339858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49461 (local address 127.0.98.62:33549)
0925 13:17:29.339999 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:29.340001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:29.340164 (+ 163us) server_negotiation.cc:407] Connection header received
0925 13:17:29.340861 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:29.340862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:29.340901 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:29.340972 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:29.341889 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.342407 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.343941 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.344157 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.344223 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.347125 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:29.347126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:29.347127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:29.347138 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:29.348393 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:29.348761 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:29.348763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:29.348763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:29.348817 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:29.349022 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:29.349023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:29.349023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:29.349140 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:29.349194 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:29.349453 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 13:17:29.349566 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260925 13:17:29.349828 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.339825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33549 (local address 127.0.98.2:49461)
0925 13:17:29.340015 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:29.340069 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:29.340736 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:29.341072 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:29.341076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:29.341453 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:29.341799 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.341807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.342480 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.342480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.343855 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.343858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.344251 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.344252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.344373 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.345379 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:29.345391 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:29.347035 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:29.348467 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:29.348474 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:29.348476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:29.348697 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:29.348870 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:29.348871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:29.348871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:29.348966 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:29.349194 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:29.349201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:29.349337 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 13:17:29.349427 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 13:17:29.349624 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260925 13:17:29.350240 1852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33549
I20260925 13:17:29.350313 1852 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:29.350477 1852 heartbeater.cc:511] Master 127.0.98.62:33549 requested a full tablet report, sending...
I20260925 13:17:29.350837 1507 ts_manager.cc:194] Registered new tserver with Master: a66ecc6d04e74e739b72776e786ad234 (127.0.98.2:43455)
I20260925 13:17:29.351300 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49461
I20260925 13:17:29.352919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 1720
I20260925 13:17:29.353008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:17:29.353173 392 external_mini_cluster.cc:1428] Setting key 1ee483bbd7f3adf9fa3158614b5312ba
I20260925 13:17:29.356560 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.351192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36545 (local address 127.0.98.62:33549)
0925 13:17:29.351228 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:29.351229 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:29.351302 (+ 73us) server_negotiation.cc:407] Connection header received
0925 13:17:29.351366 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:29.351367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:29.351377 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:29.351432 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:29.351761 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.352314 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.352939 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.353235 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.353301 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.354152 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:29.354153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:29.354154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:29.354161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:29.355423 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:29.355820 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:29.355821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:29.355822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:29.355888 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:29.356071 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:29.356072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:29.356072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:29.356214 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:29.356269 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:29.356410 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 13:17:29.356494 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 13:17:29.356581 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.351136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33549 (local address 127.0.98.2:36545)
0925 13:17:29.351256 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:29.351265 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:29.351314 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:29.351478 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:29.351479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:29.351640 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:29.351693 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.351695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.352386 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.352387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.352878 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.352879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.353299 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.353300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.353362 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.353815 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:29.353820 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:29.354080 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:29.355500 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:29.355502 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:29.355502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:29.355726 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:29.355956 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:29.355957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:29.355957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:29.356025 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:29.356266 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:29.356266 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:29.356312 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 13:17:29.356381 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 13:17:29.356482 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:17:29.375900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33549
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:29.456225 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:29.456430 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:29.456453 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:29.457578 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:29.457623 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:29.457641 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:17:29.457657 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:17:29.459161 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:29.459365 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:29.459571 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:29.464346 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:29.464375 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:29.464349 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:29.464720 1861 server_base.cc:1034] running on GCE node
I20260925 13:17:29.464901 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:29.465106 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:29.466229 1861 hybrid_clock.cc:648] HybridClock initialized: now 1790342249466213 us; error 29 us; skew 500 ppm
Sep 25 13:17:29 dist-test-slave-t8d9 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:29.468604 1861 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:17:29.468992 1861 webserver.cc:460] Webserver started at http://127.0.98.3:45851/ using document root <none> and password file <none>
I20260925 13:17:29.469163 1861 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:29.469202 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:29.469293 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:29.470346 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "6599055bc62a4825a72cc38641bf0a96"
format_stamp: "Formatted at 2026-09-25 13:17:29 on dist-test-slave-t8d9"
server_key: "d23581224cd74bd86ecb578bc61f3c7a"
server_key_iv: "00f0d529f1765af85261fc428a9fa0cd"
server_key_version: "encryptionkey@0"
I20260925 13:17:29.470651 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6599055bc62a4825a72cc38641bf0a96"
format_stamp: "Formatted at 2026-09-25 13:17:29 on dist-test-slave-t8d9"
server_key: "d23581224cd74bd86ecb578bc61f3c7a"
server_key_iv: "00f0d529f1765af85261fc428a9fa0cd"
server_key_version: "encryptionkey@0"
I20260925 13:17:29.471884 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:17:29.472520 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:29.472697 1861 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:29.472779 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "6599055bc62a4825a72cc38641bf0a96"
format_stamp: "Formatted at 2026-09-25 13:17:29 on dist-test-slave-t8d9"
server_key: "d23581224cd74bd86ecb578bc61f3c7a"
server_key_iv: "00f0d529f1765af85261fc428a9fa0cd"
server_key_version: "encryptionkey@0"
I20260925 13:17:29.472856 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:29.485065 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:29.487532 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:29.487665 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:29.488186 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:29.488230 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:29.488252 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:29.488265 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:29.491211 1711 heartbeater.cc:503] Master 127.0.98.62:33549 was elected leader, sending a full tablet report...
I20260925 13:17:29.494807 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33829
I20260925 13:17:29.494877 1991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33829 every 8 connection(s)
I20260925 13:17:29.495187 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:17:29 dist-test-slave-t8d9 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:29.498026 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.488716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43287 (local address 127.0.98.62:33549)
0925 13:17:29.488760 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:29.488762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:29.488998 (+ 236us) server_negotiation.cc:407] Connection header received
0925 13:17:29.489505 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:29.489507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:29.489522 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:29.489620 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:29.490495 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.491299 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.492800 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.493110 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.493192 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.495716 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:29.495718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:29.495718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:29.495726 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:29.496786 (+ 1060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:29.497128 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:29.497128 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:29.497129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:29.497174 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:29.497350 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:29.497350 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:29.497351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:29.497445 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:29.497490 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:29.497897 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 13:17:29.497963 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:17:29.498185 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.488731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33549 (local address 127.0.98.3:43287)
0925 13:17:29.488876 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:29.488915 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:29.489406 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:29.489742 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:29.489748 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:29.490089 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:29.490383 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.490389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.491404 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.491405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.492703 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.492706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.493195 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.493195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.493301 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.493910 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:29.493921 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:29.495637 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:29.496851 (+ 1214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:29.496856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:29.496858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:29.497070 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:29.497219 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:29.497219 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:29.497219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:29.497310 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:29.497605 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:29.497612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:29.497766 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 13:17:29.497867 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 13:17:29.497999 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260925 13:17:29.498677 1992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33549
I20260925 13:17:29.498757 1992 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:29.498924 1992 heartbeater.cc:511] Master 127.0.98.62:33549 requested a full tablet report, sending...
I20260925 13:17:29.499305 1507 ts_manager.cc:194] Registered new tserver with Master: 6599055bc62a4825a72cc38641bf0a96 (127.0.98.3:33829)
I20260925 13:17:29.499727 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43287
I20260925 13:17:29.501114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 1861
I20260925 13:17:29.501194 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:17:29.501345 392 external_mini_cluster.cc:1428] Setting key f81fab0866fd61f244e17da1ec351650
I20260925 13:17:29.502178 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:17:29.505236 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.499653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43047 (local address 127.0.98.62:33549)
0925 13:17:29.499692 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:29.499694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:29.499788 (+ 94us) server_negotiation.cc:407] Connection header received
0925 13:17:29.499901 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:29.499901 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:29.499909 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:29.499946 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:29.500302 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.500812 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.501631 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.501901 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.501963 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.502979 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:29.502980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:29.502980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:29.502986 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:29.504242 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:29.504555 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:29.504556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:29.504556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:29.504613 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:29.504773 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:29.504774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:29.504774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:29.504879 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:29.504932 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:29.505064 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 13:17:29.505139 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:17:29.505246 1998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.499578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33549 (local address 127.0.98.3:43047)
0925 13:17:29.499729 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:29.499740 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:29.499799 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:29.499970 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:29.499971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:29.500146 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:29.500203 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.500204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.500903 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.500904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.501501 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.501503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.501967 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.501968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.502032 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.502512 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:29.502518 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:29.502899 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:29.504318 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:29.504319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:29.504319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:29.504491 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:29.504664 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:29.504665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:29.504665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:29.504725 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:29.504931 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:29.504932 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:29.504977 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 13:17:29.505035 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 13:17:29.505139 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
Sep 25 13:17:29 dist-test-slave-t8d9 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342249, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 13:17:29 dist-test-slave-t8d9 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342249, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:29.523873 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.517236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.0.98.62:33549)
0925 13:17:29.517300 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:29.517302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:29.517312 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:17:29.517339 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:29.517340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:29.517351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:29.517394 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:29.517973 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.518494 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.519231 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.519412 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.519492 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.521652 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:29.521654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:29.521654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:29.521662 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:29.522850 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:29.523174 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:29.523176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:29.523176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:29.523236 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:29.523396 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:29.523396 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:29.523397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:29.523508 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:29.523593 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:29.523699 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 13:17:29.523780 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:17:29.525377 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:17:29.526985 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:29.533334 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.529261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33829 (local address 127.0.0.1:50100)
0925 13:17:29.529500 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:29.529616 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:29.529687 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:29.530025 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:29.530028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:29.530030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:29.530130 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.530132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.530977 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.530977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.532138 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.532141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.533033 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.533034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.533111 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.533134 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 13:17:29.533197 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 13:17:29.533249 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":5,"thread_start_us":105,"threads_started":1}
I20260925 13:17:29.533371 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.529570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.0.98.3:33829)
0925 13:17:29.529777 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:29.529782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:29.529792 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:17:29.529829 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:29.529830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:29.529843 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:29.529906 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:29.530210 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.530900 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.532218 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.532953 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.533025 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.533101 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:29.533230 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 13:17:29.533271 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260925 13:17:29.533421 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.529265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38825 (local address 127.0.0.1:41534)
0925 13:17:29.529785 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:29.529795 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:29.529831 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:29.530089 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:29.530089 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:29.530091 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:29.530143 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.530145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.531421 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.531422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.532420 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.532423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.533238 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.533239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.533285 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.533289 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:29.533335 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 13:17:29.533368 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":25,"thread_start_us":114,"threads_started":1}
I20260925 13:17:29.533674 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.529473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.0.98.1:38825)
0925 13:17:29.529936 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:29.529940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:29.529948 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:17:29.529978 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:29.529980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:29.529993 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:29.530215 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:29.530271 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.531338 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.532482 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.533169 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.533232 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.533284 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:29.533347 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 13:17:29.533385 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":47,"threads_started":1}
I20260925 13:17:29.533813 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.529261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43455 (local address 127.0.0.1:49298)
0925 13:17:29.529617 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:29.529627 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:29.529687 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:29.530025 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:29.530030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:29.530033 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:29.530176 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.530177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.531449 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.531450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.532364 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.532366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.533600 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.533601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.533667 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.533671 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:29.533716 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 13:17:29.533750 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":88,"thread_start_us":106,"threads_started":1}
I20260925 13:17:29.534049 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.529438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.0.98.2:43455)
0925 13:17:29.529820 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:29.529825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:29.529834 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:29.529870 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:29.529872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:29.529887 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:29.529956 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:29.530245 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.531338 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.532443 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.532956 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.533021 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.533100 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:29.533913 (+ 813us) server_negotiation.cc:299] Negotiation successful
0925 13:17:29.533956 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":74,"threads_started":1}
I20260925 13:17:29.534162 1926 tablet_service.cc:1467] Processing CreateTablet for tablet c74fd907d1004ddc9651b13d9fc22940 (DEFAULT_TABLE table=test-table [id=1fa8b2343b794942a3268c5715de5b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:29.534395 1645 tablet_service.cc:1467] Processing CreateTablet for tablet c74fd907d1004ddc9651b13d9fc22940 (DEFAULT_TABLE table=test-table [id=1fa8b2343b794942a3268c5715de5b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:29.534564 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74fd907d1004ddc9651b13d9fc22940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:29.534701 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74fd907d1004ddc9651b13d9fc22940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:29.534591 1786 tablet_service.cc:1467] Processing CreateTablet for tablet c74fd907d1004ddc9651b13d9fc22940 (DEFAULT_TABLE table=test-table [id=1fa8b2343b794942a3268c5715de5b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:29.534881 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74fd907d1004ddc9651b13d9fc22940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:29.537276 2016 tablet_bootstrap.cc:492] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600: Bootstrap starting.
I20260925 13:17:29.537914 2016 tablet_bootstrap.cc:654] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:29.538126 2016 log.cc:824] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:29.538404 2017 tablet_bootstrap.cc:492] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96: Bootstrap starting.
I20260925 13:17:29.538793 2016 tablet_bootstrap.cc:492] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600: No bootstrap required, opened a new log
I20260925 13:17:29.538867 2016 ts_tablet_manager.cc:1397] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 13:17:29.539021 2017 tablet_bootstrap.cc:654] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:29.539341 2017 log.cc:824] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:29.539734 2018 tablet_bootstrap.cc:492] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234: Bootstrap starting.
I20260925 13:17:29.540073 2017 tablet_bootstrap.cc:492] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96: No bootstrap required, opened a new log
I20260925 13:17:29.540138 2017 ts_tablet_manager.cc:1397] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:17:29.540392 2018 tablet_bootstrap.cc:654] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:29.540719 2018 log.cc:824] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:29.541508 2018 tablet_bootstrap.cc:492] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234: No bootstrap required, opened a new log
I20260925 13:17:29.541626 2018 ts_tablet_manager.cc:1397] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:17:29.541497 2016 raft_consensus.cc:348] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.541710 2016 raft_consensus.cc:374] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:29.541757 2016 raft_consensus.cc:729] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1bb58fa8f947f1b2ec7718f60a5600, State: Initialized, Role: FOLLOWER
I20260925 13:17:29.541873 2016 consensus_queue.cc:260] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.542109 2016 ts_tablet_manager.cc:1428] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 13:17:29.542083 2017 raft_consensus.cc:348] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.542256 2017 raft_consensus.cc:374] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:29.542299 2017 raft_consensus.cc:729] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6599055bc62a4825a72cc38641bf0a96, State: Initialized, Role: FOLLOWER
I20260925 13:17:29.542407 2017 consensus_queue.cc:260] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.542666 2017 ts_tablet_manager.cc:1428] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 13:17:29.542806 1992 heartbeater.cc:503] Master 127.0.98.62:33549 was elected leader, sending a full tablet report...
I20260925 13:17:29.543581 2018 raft_consensus.cc:348] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.543740 2018 raft_consensus.cc:374] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:29.543782 2018 raft_consensus.cc:729] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66ecc6d04e74e739b72776e786ad234, State: Initialized, Role: FOLLOWER
I20260925 13:17:29.543884 2018 consensus_queue.cc:260] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.544086 2018 ts_tablet_manager.cc:1428] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:17:29.544135 1852 heartbeater.cc:503] Master 127.0.98.62:33549 was elected leader, sending a full tablet report...
W20260925 13:17:29.597710 1853 tablet.cc:2369] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:17:29.626240 2023 raft_consensus.cc:482] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:29.626370 2023 raft_consensus.cc:504] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.626785 2023 leader_election.cc:288] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a66ecc6d04e74e739b72776e786ad234 (127.0.98.2:43455), 8c1bb58fa8f947f1b2ec7718f60a5600 (127.0.98.1:38825)
I20260925 13:17:29.629631 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.627107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33955 (local address 127.0.98.1:38825)
0925 13:17:29.627167 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:29.627169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:29.627246 (+ 77us) server_negotiation.cc:407] Connection header received
0925 13:17:29.627304 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:29.627305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:29.627315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:29.627354 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:29.627487 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.627995 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.628955 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.629303 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.629361 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.629372 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:29.629502 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 13:17:29.629566 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:17:29.629644 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.626998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38825 (local address 127.0.98.3:33955)
0925 13:17:29.627173 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:29.627182 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:29.627243 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:29.627364 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:29.627365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:29.627366 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:29.627431 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.627432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.628094 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.628094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.628861 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.628863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.629375 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.629376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.629441 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.629445 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:29.629489 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 13:17:29.629528 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":115,"threads_started":1}
I20260925 13:17:29.629650 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.627130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60755 (local address 127.0.98.2:43455)
0925 13:17:29.627167 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:29.627169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:29.627177 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:17:29.627202 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:29.627203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:29.627213 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:29.627254 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:29.627403 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.627888 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.628951 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.629320 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.629378 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.629389 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:29.629521 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 13:17:29.629589 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 13:17:29.629663 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.626978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43455 (local address 127.0.98.3:60755)
0925 13:17:29.627017 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:29.627029 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:29.627076 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:29.627281 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:29.627282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:29.627284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:29.627341 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.627343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.628008 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.628009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.628861 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.628863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.629392 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.629393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.629454 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.629457 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:29.629500 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:17:29.629564 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":49}
I20260925 13:17:29.630203 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74fd907d1004ddc9651b13d9fc22940" candidate_uuid: "6599055bc62a4825a72cc38641bf0a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" is_pre_election: true
I20260925 13:17:29.630203 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74fd907d1004ddc9651b13d9fc22940" candidate_uuid: "6599055bc62a4825a72cc38641bf0a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66ecc6d04e74e739b72776e786ad234" is_pre_election: true
I20260925 13:17:29.630367 1806 raft_consensus.cc:2454] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6599055bc62a4825a72cc38641bf0a96 in term 0.
I20260925 13:17:29.630367 1665 raft_consensus.cc:2454] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6599055bc62a4825a72cc38641bf0a96 in term 0.
I20260925 13:17:29.630600 1877 leader_election.cc:302] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6599055bc62a4825a72cc38641bf0a96, 8c1bb58fa8f947f1b2ec7718f60a5600; no voters:
I20260925 13:17:29.630716 2023 raft_consensus.cc:2781] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:29.630771 2023 raft_consensus.cc:482] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:29.630786 2023 raft_consensus.cc:3037] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:29.631649 2023 raft_consensus.cc:504] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.631764 2023 leader_election.cc:288] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [CANDIDATE]: Term 1 election: Requested vote from peers a66ecc6d04e74e739b72776e786ad234 (127.0.98.2:43455), 8c1bb58fa8f947f1b2ec7718f60a5600 (127.0.98.1:38825)
I20260925 13:17:29.631927 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74fd907d1004ddc9651b13d9fc22940" candidate_uuid: "6599055bc62a4825a72cc38641bf0a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600"
I20260925 13:17:29.631938 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74fd907d1004ddc9651b13d9fc22940" candidate_uuid: "6599055bc62a4825a72cc38641bf0a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66ecc6d04e74e739b72776e786ad234"
I20260925 13:17:29.631999 1665 raft_consensus.cc:3037] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:29.632010 1806 raft_consensus.cc:3037] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:29.632685 1806 raft_consensus.cc:2454] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6599055bc62a4825a72cc38641bf0a96 in term 1.
I20260925 13:17:29.632690 1665 raft_consensus.cc:2454] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6599055bc62a4825a72cc38641bf0a96 in term 1.
I20260925 13:17:29.632844 1877 leader_election.cc:302] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6599055bc62a4825a72cc38641bf0a96, 8c1bb58fa8f947f1b2ec7718f60a5600; no voters:
I20260925 13:17:29.632925 2023 raft_consensus.cc:2781] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:29.633068 2023 raft_consensus.cc:686] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [term 1 LEADER]: Becoming Leader. State: Replica: 6599055bc62a4825a72cc38641bf0a96, State: Running, Role: LEADER
I20260925 13:17:29.633157 2023 consensus_queue.cc:237] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.634084 1506 catalog_manager.cc:5495] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6599055bc62a4825a72cc38641bf0a96 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6599055bc62a4825a72cc38641bf0a96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } health_report { overall_health: HEALTHY } } }
I20260925 13:17:29.650785 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.648301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.0.98.3:33829)
0925 13:17:29.648357 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:29.648359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:29.648370 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:17:29.648507 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:29.648508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:29.648528 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:29.648596 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:29.648765 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.649241 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.650015 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.650268 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.650320 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.650493 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:29.650597 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:29.650653 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:29.650696 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 13:17:29.650712 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:17:29.652684 1665 raft_consensus.cc:1261] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 1 FOLLOWER]: Refusing update from remote peer 6599055bc62a4825a72cc38641bf0a96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:29.652684 1806 raft_consensus.cc:1261] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234 [term 1 FOLLOWER]: Refusing update from remote peer 6599055bc62a4825a72cc38641bf0a96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:29.653038 2026 consensus_queue.cc:1035] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:29.653193 2023 consensus_queue.cc:1035] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:29.655764 2031 mvcc.cc:205] Tried to move back new op lower bound from 7333241854574616576 to 7333241854498201600. Current Snapshot: MvccSnapshot[applied={T|T < 7333241854574616576}]
I20260925 13:17:29.656059 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333241854574616576 to 7333241854498201600. Current Snapshot: MvccSnapshot[applied={T|T < 7333241854574616576}]
I20260925 13:17:29.659243 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:29.659346 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:29.660100 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4: Sending DeleteTablet for 3 replicas of tablet c74fd907d1004ddc9651b13d9fc22940
I20260925 13:17:29.660408 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet c74fd907d1004ddc9651b13d9fc22940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50100
I20260925 13:17:29.660547 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet c74fd907d1004ddc9651b13d9fc22940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49298
I20260925 13:17:29.660753 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet c74fd907d1004ddc9651b13d9fc22940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41534
I20260925 13:17:29.660874 2038 tablet_replica.cc:330] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96: stopping tablet replica
I20260925 13:17:29.660969 2038 raft_consensus.cc:2229] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:17:29.661041 2039 tablet_replica.cc:330] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234: stopping tablet replica
I20260925 13:17:29.660954 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:17:29.661121 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:29.661139 2038 raft_consensus.cc:2258] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:29.661348 2039 raft_consensus.cc:2229] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:29.661412 2040 tablet_replica.cc:330] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600: stopping tablet replica
I20260925 13:17:29.661581 2039 raft_consensus.cc:2258] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:29.661628 2040 raft_consensus.cc:2229] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:29.661811 2040 raft_consensus.cc:2258] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:29.662194 2039 ts_tablet_manager.cc:1905] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:29.662689 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 177e6a084a284a1faef4f8ec4d3b30ec (DEFAULT_TABLE table=test-table [id=e65c1dda42574253b9e72a6e2da5202d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:29.662758 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 177e6a084a284a1faef4f8ec4d3b30ec (DEFAULT_TABLE table=test-table [id=e65c1dda42574253b9e72a6e2da5202d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:29.662815 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177e6a084a284a1faef4f8ec4d3b30ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:29.662840 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177e6a084a284a1faef4f8ec4d3b30ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:29.663923 2039 ts_tablet_manager.cc:1918] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:29.663990 2039 log.cc:1196] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/c74fd907d1004ddc9651b13d9fc22940
I20260925 13:17:29.664100 2016 tablet_bootstrap.cc:492] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600: Bootstrap starting.
I20260925 13:17:29.664163 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 177e6a084a284a1faef4f8ec4d3b30ec (DEFAULT_TABLE table=test-table [id=e65c1dda42574253b9e72a6e2da5202d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:29.664225 2039 ts_tablet_manager.cc:1939] T c74fd907d1004ddc9651b13d9fc22940 P a66ecc6d04e74e739b72776e786ad234: Deleting consensus metadata
I20260925 13:17:29.664270 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177e6a084a284a1faef4f8ec4d3b30ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:29.664490 2016 tablet_bootstrap.cc:654] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:29.662688 2040 ts_tablet_manager.cc:1905] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:29.665042 2016 tablet_bootstrap.cc:492] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600: No bootstrap required, opened a new log
I20260925 13:17:29.665088 2016 ts_tablet_manager.cc:1397] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 13:17:29.665241 2016 raft_consensus.cc:348] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.665308 2016 raft_consensus.cc:374] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:29.665328 2016 raft_consensus.cc:729] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1bb58fa8f947f1b2ec7718f60a5600, State: Initialized, Role: FOLLOWER
I20260925 13:17:29.665369 2016 consensus_queue.cc:260] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.665602 2018 tablet_bootstrap.cc:492] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234: Bootstrap starting.
I20260925 13:17:29.665925 2016 ts_tablet_manager.cc:1428] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 13:17:29.665977 2018 tablet_bootstrap.cc:654] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:29.666082 2040 ts_tablet_manager.cc:1918] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:29.666157 2040 log.cc:1196] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/wal/wals/c74fd907d1004ddc9651b13d9fc22940
I20260925 13:17:29.666308 2017 tablet_bootstrap.cc:492] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96: Bootstrap starting.
I20260925 13:17:29.666536 2018 tablet_bootstrap.cc:492] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234: No bootstrap required, opened a new log
I20260925 13:17:29.666576 2018 ts_tablet_manager.cc:1397] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:29.666715 2017 tablet_bootstrap.cc:654] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:29.666723 2018 raft_consensus.cc:348] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.666798 2018 raft_consensus.cc:374] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:29.666819 2018 raft_consensus.cc:729] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66ecc6d04e74e739b72776e786ad234, State: Initialized, Role: FOLLOWER
I20260925 13:17:29.666864 2018 consensus_queue.cc:260] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.666947 2018 ts_tablet_manager.cc:1428] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:29.667202 2040 ts_tablet_manager.cc:1939] T c74fd907d1004ddc9651b13d9fc22940 P 8c1bb58fa8f947f1b2ec7718f60a5600: Deleting consensus metadata
I20260925 13:17:29.667371 1492 catalog_manager.cc:4841] TS a66ecc6d04e74e739b72776e786ad234 (127.0.98.2:43455): tablet c74fd907d1004ddc9651b13d9fc22940 (table test-table [id=1fa8b2343b794942a3268c5715de5b2d]) successfully deleted
I20260925 13:17:29.667802 1491 catalog_manager.cc:4841] TS 8c1bb58fa8f947f1b2ec7718f60a5600 (127.0.98.1:38825): tablet c74fd907d1004ddc9651b13d9fc22940 (table test-table [id=1fa8b2343b794942a3268c5715de5b2d]) successfully deleted
I20260925 13:17:29.667963 2038 ts_tablet_manager.cc:1905] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:29.669149 2038 ts_tablet_manager.cc:1918] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:29.669216 2038 log.cc:1196] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/c74fd907d1004ddc9651b13d9fc22940
I20260925 13:17:29.669416 2038 ts_tablet_manager.cc:1939] T c74fd907d1004ddc9651b13d9fc22940 P 6599055bc62a4825a72cc38641bf0a96: Deleting consensus metadata
I20260925 13:17:29.669823 1493 catalog_manager.cc:4841] TS 6599055bc62a4825a72cc38641bf0a96 (127.0.98.3:33829): tablet c74fd907d1004ddc9651b13d9fc22940 (table test-table [id=1fa8b2343b794942a3268c5715de5b2d]) successfully deleted
I20260925 13:17:29.670019 2017 tablet_bootstrap.cc:492] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96: No bootstrap required, opened a new log
I20260925 13:17:29.670070 2017 ts_tablet_manager.cc:1397] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 13:17:29.670217 2017 raft_consensus.cc:348] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.670293 2017 raft_consensus.cc:374] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:29.670313 2017 raft_consensus.cc:729] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6599055bc62a4825a72cc38641bf0a96, State: Initialized, Role: FOLLOWER
I20260925 13:17:29.670367 2017 consensus_queue.cc:260] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.670455 2017 ts_tablet_manager.cc:1428] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260925 13:17:29.737479 1712 tablet.cc:2369] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:17:29.745702 1993 tablet.cc:2369] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:17:29.908983 2022 raft_consensus.cc:482] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:29.909102 2022 raft_consensus.cc:504] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.909375 2022 leader_election.cc:288] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a66ecc6d04e74e739b72776e786ad234 (127.0.98.2:43455), 6599055bc62a4825a72cc38641bf0a96 (127.0.98.3:33829)
I20260925 13:17:29.912503 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.909653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39525 (local address 127.0.98.2:43455)
0925 13:17:29.909723 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:29.909724 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:29.910008 (+ 284us) server_negotiation.cc:407] Connection header received
0925 13:17:29.910037 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:29.910038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:29.910047 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:29.910102 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:29.910274 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.910771 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.911821 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.912183 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.912241 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.912251 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:29.912399 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 13:17:29.912435 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 13:17:29.912503 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.909653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43273 (local address 127.0.98.3:33829)
0925 13:17:29.909718 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:29.909720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:29.909840 (+ 120us) server_negotiation.cc:407] Connection header received
0925 13:17:29.909916 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:29.909918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:29.909929 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:29.910000 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:29.910179 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.910704 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.911816 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.912178 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.912240 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.912252 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:29.912391 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 13:17:29.912433 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 13:17:29.912536 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.909611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43455 (local address 127.0.98.1:39525)
0925 13:17:29.909912 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:29.909922 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:29.909995 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:29.910109 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:29.910110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:29.910111 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:29.910178 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.910179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.910874 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.910875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.911728 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.911731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.912276 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.912277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.912332 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.912335 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:29.912394 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 13:17:29.912434 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":24,"thread_start_us":79,"threads_started":1}
I20260925 13:17:29.912540 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.909612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33829 (local address 127.0.98.1:43273)
0925 13:17:29.909772 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:29.909782 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:29.909842 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:29.910012 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:29.910013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:29.910016 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:29.910094 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.910096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.910817 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.910818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.911729 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.911731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.912260 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.912261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.912326 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.912329 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:29.912372 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:17:29.912417 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":45,"spinlock_wait_cycles":896,"thread_start_us":82,"threads_started":1}
I20260925 13:17:29.912792 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177e6a084a284a1faef4f8ec4d3b30ec" candidate_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66ecc6d04e74e739b72776e786ad234" is_pre_election: true
I20260925 13:17:29.912905 1806 raft_consensus.cc:2454] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1bb58fa8f947f1b2ec7718f60a5600 in term 0.
I20260925 13:17:29.913012 1938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177e6a084a284a1faef4f8ec4d3b30ec" candidate_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6599055bc62a4825a72cc38641bf0a96" is_pre_election: true
I20260925 13:17:29.913089 1596 leader_election.cc:302] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c1bb58fa8f947f1b2ec7718f60a5600, a66ecc6d04e74e739b72776e786ad234; no voters:
I20260925 13:17:29.913159 1938 raft_consensus.cc:2454] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1bb58fa8f947f1b2ec7718f60a5600 in term 0.
I20260925 13:17:29.913203 2022 raft_consensus.cc:2781] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:29.913237 2022 raft_consensus.cc:482] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:29.913254 2022 raft_consensus.cc:3037] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:29.913982 2022 raft_consensus.cc:504] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.914121 2022 leader_election.cc:288] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [CANDIDATE]: Term 1 election: Requested vote from peers a66ecc6d04e74e739b72776e786ad234 (127.0.98.2:43455), 6599055bc62a4825a72cc38641bf0a96 (127.0.98.3:33829)
I20260925 13:17:29.914248 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177e6a084a284a1faef4f8ec4d3b30ec" candidate_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66ecc6d04e74e739b72776e786ad234"
I20260925 13:17:29.914316 1806 raft_consensus.cc:3037] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:29.914275 1938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177e6a084a284a1faef4f8ec4d3b30ec" candidate_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6599055bc62a4825a72cc38641bf0a96"
I20260925 13:17:29.914348 1938 raft_consensus.cc:3037] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:29.914866 1806 raft_consensus.cc:2454] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1bb58fa8f947f1b2ec7718f60a5600 in term 1.
I20260925 13:17:29.915024 1938 raft_consensus.cc:2454] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1bb58fa8f947f1b2ec7718f60a5600 in term 1.
I20260925 13:17:29.915025 1596 leader_election.cc:302] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c1bb58fa8f947f1b2ec7718f60a5600, a66ecc6d04e74e739b72776e786ad234; no voters:
I20260925 13:17:29.915122 2022 raft_consensus.cc:2781] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:29.915277 2022 raft_consensus.cc:686] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 1 LEADER]: Becoming Leader. State: Replica: 8c1bb58fa8f947f1b2ec7718f60a5600, State: Running, Role: LEADER
I20260925 13:17:29.915345 2022 consensus_queue.cc:237] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.916055 1506 catalog_manager.cc:5495] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c1bb58fa8f947f1b2ec7718f60a5600 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } health_report { overall_health: UNKNOWN } } }
Sep 25 13:17:29 dist-test-slave-t8d9 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:29.977038 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.970240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33549 (local address 127.0.0.1:60616)
0925 13:17:29.970420 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:29.970430 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:29.970503 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:29.970631 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:29.970632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:29.970927 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:29.970982 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.970983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.971696 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.971697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.972229 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:29.972231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.972604 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.972606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:29.972666 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.973191 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:29.973196 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:29.974673 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:29.976037 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:29.976065 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:29.976067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:29.976288 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:29.976433 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:29.976434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:29.976434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:29.976501 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:29.976704 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:29.976712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:29.976760 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 13:17:29.976826 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 13:17:29.976935 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
I20260925 13:17:29.977079 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:29.970335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.0.98.62:33549)
0925 13:17:29.970421 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:29.970422 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:29.970491 (+ 69us) server_negotiation.cc:407] Connection header received
0925 13:17:29.970558 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:29.970559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:29.970567 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:29.970618 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:29.971092 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.971607 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.972311 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:29.972501 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:29.972578 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:29.974750 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:29.974751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:29.974751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:29.974757 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:29.975955 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:29.976347 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:29.976347 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:29.976348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:29.976390 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:29.976543 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:29.976543 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:29.976543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:29.976636 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:29.976687 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:29.976850 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 13:17:29.976976 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":41}
I20260925 13:17:29.978894 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60616:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 13:17:29.979072 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:29.980273 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 8390c6918483471c8727f8ee73d006f0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4844b539a29b4d1f8d32c6cc849abca6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:17:29.980280 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 8390c6918483471c8727f8ee73d006f0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4844b539a29b4d1f8d32c6cc849abca6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:17:29.980404 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8390c6918483471c8727f8ee73d006f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:29.980417 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8390c6918483471c8727f8ee73d006f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:29.981482 2018 tablet_bootstrap.cc:492] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234: Bootstrap starting.
I20260925 13:17:29.981660 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 8390c6918483471c8727f8ee73d006f0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4844b539a29b4d1f8d32c6cc849abca6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:17:29.981783 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8390c6918483471c8727f8ee73d006f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:29.981899 2018 tablet_bootstrap.cc:654] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:29.982213 2016 tablet_bootstrap.cc:492] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600: Bootstrap starting.
I20260925 13:17:29.982466 2018 tablet_bootstrap.cc:492] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234: No bootstrap required, opened a new log
I20260925 13:17:29.982501 2018 ts_tablet_manager.cc:1397] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:29.982573 2016 tablet_bootstrap.cc:654] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:29.982609 2018 raft_consensus.cc:348] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.982664 2018 raft_consensus.cc:374] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:29.982677 2018 raft_consensus.cc:729] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66ecc6d04e74e739b72776e786ad234, State: Initialized, Role: FOLLOWER
I20260925 13:17:29.982714 2018 consensus_queue.cc:260] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.982796 2018 ts_tablet_manager.cc:1428] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:29.982841 2024 tablet_replica.cc:439] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } } }
I20260925 13:17:29.982915 2024 tablet_replica.cc:442] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:29.983678 2017 tablet_bootstrap.cc:492] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96: Bootstrap starting.
I20260925 13:17:29.983978 2017 tablet_bootstrap.cc:654] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:29.983996 2016 tablet_bootstrap.cc:492] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600: No bootstrap required, opened a new log
I20260925 13:17:29.984028 2016 ts_tablet_manager.cc:1397] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:17:29.984143 2016 raft_consensus.cc:348] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.984216 2016 raft_consensus.cc:374] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:29.984243 2016 raft_consensus.cc:729] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1bb58fa8f947f1b2ec7718f60a5600, State: Initialized, Role: FOLLOWER
I20260925 13:17:29.984287 2016 consensus_queue.cc:260] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.984397 2016 ts_tablet_manager.cc:1428] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:29.984438 2017 tablet_bootstrap.cc:492] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96: No bootstrap required, opened a new log
I20260925 13:17:29.984448 2022 tablet_replica.cc:439] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } } }
I20260925 13:17:29.984488 2017 ts_tablet_manager.cc:1397] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 13:17:29.984517 2022 tablet_replica.cc:442] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:29.984588 2017 raft_consensus.cc:348] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.984650 2017 raft_consensus.cc:374] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:29.984668 2017 raft_consensus.cc:729] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6599055bc62a4825a72cc38641bf0a96, State: Initialized, Role: FOLLOWER
I20260925 13:17:29.984712 2017 consensus_queue.cc:260] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:29.984800 2017 ts_tablet_manager.cc:1428] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:29.984820 2026 tablet_replica.cc:439] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } } }
I20260925 13:17:29.984908 2026 tablet_replica.cc:442] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:30.200124 2022 raft_consensus.cc:482] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:30.200233 2022 raft_consensus.cc:504] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:30.200392 2022 leader_election.cc:288] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a66ecc6d04e74e739b72776e786ad234 (127.0.98.2:43455), 6599055bc62a4825a72cc38641bf0a96 (127.0.98.3:33829)
I20260925 13:17:30.200611 1938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390c6918483471c8727f8ee73d006f0" candidate_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6599055bc62a4825a72cc38641bf0a96" is_pre_election: true
I20260925 13:17:30.200646 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390c6918483471c8727f8ee73d006f0" candidate_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66ecc6d04e74e739b72776e786ad234" is_pre_election: true
I20260925 13:17:30.200727 1806 raft_consensus.cc:2454] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1bb58fa8f947f1b2ec7718f60a5600 in term 0.
I20260925 13:17:30.200727 1938 raft_consensus.cc:2454] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1bb58fa8f947f1b2ec7718f60a5600 in term 0.
I20260925 13:17:30.200873 1596 leader_election.cc:302] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c1bb58fa8f947f1b2ec7718f60a5600, a66ecc6d04e74e739b72776e786ad234; no voters:
I20260925 13:17:30.200987 2022 raft_consensus.cc:2781] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:30.201026 2022 raft_consensus.cc:482] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:30.201046 2022 raft_consensus.cc:3037] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:30.201745 2022 raft_consensus.cc:504] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:30.201894 2022 leader_election.cc:288] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [CANDIDATE]: Term 1 election: Requested vote from peers a66ecc6d04e74e739b72776e786ad234 (127.0.98.2:43455), 6599055bc62a4825a72cc38641bf0a96 (127.0.98.3:33829)
I20260925 13:17:30.202060 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390c6918483471c8727f8ee73d006f0" candidate_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66ecc6d04e74e739b72776e786ad234"
I20260925 13:17:30.202140 1806 raft_consensus.cc:3037] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:30.202776 1806 raft_consensus.cc:2454] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1bb58fa8f947f1b2ec7718f60a5600 in term 1.
I20260925 13:17:30.202889 1938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390c6918483471c8727f8ee73d006f0" candidate_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6599055bc62a4825a72cc38641bf0a96"
I20260925 13:17:30.202937 1596 leader_election.cc:302] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c1bb58fa8f947f1b2ec7718f60a5600, a66ecc6d04e74e739b72776e786ad234; no voters:
I20260925 13:17:30.202972 1938 raft_consensus.cc:3037] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:30.203027 2022 raft_consensus.cc:2781] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:30.203088 2022 raft_consensus.cc:686] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 1 LEADER]: Becoming Leader. State: Replica: 8c1bb58fa8f947f1b2ec7718f60a5600, State: Running, Role: LEADER
I20260925 13:17:30.203137 2022 consensus_queue.cc:237] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } }
I20260925 13:17:30.203415 2046 tablet_replica.cc:439] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600: TxnStatusTablet state changed. Reason: New leader 8c1bb58fa8f947f1b2ec7718f60a5600. Latest consensus state: current_term: 1 leader_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } } }
I20260925 13:17:30.203503 2046 tablet_replica.cc:442] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:17:30.203704 1938 raft_consensus.cc:2454] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1bb58fa8f947f1b2ec7718f60a5600 in term 1.
I20260925 13:17:30.203729 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 13:17:30.203848 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 13:17:30.203923 1506 catalog_manager.cc:5495] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c1bb58fa8f947f1b2ec7718f60a5600 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } health_report { overall_health: UNKNOWN } } }
I20260925 13:17:30.210011 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:30.207511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38825 (local address 127.0.0.1:41536)
0925 13:17:30.207660 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:30.207671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:30.207726 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:30.207897 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:30.207897 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:30.207899 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:17:30.207946 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:30.207948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:30.208576 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:30.208577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:30.209153 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:30.209155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:30.209489 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:30.209490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:30.209575 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:30.209594 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:17:30.209844 (+ 250us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:17:30.209852 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 13:17:30.209895 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:17:30.209943 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260925 13:17:30.210011 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:30.207548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.0.98.1:38825)
0925 13:17:30.207661 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:30.207663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:30.207706 (+ 43us) server_negotiation.cc:407] Connection header received
0925 13:17:30.207807 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:30.207807 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:30.207825 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:30.207888 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:30.207993 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:30.208508 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:30.209225 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:30.209432 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:30.209485 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:30.209684 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:30.209790 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:30.209850 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:30.209897 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 13:17:30.209919 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260925 13:17:30.211467 1938 raft_consensus.cc:1261] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96 [term 1 FOLLOWER]: Refusing update from remote peer 8c1bb58fa8f947f1b2ec7718f60a5600: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:30.211694 1806 raft_consensus.cc:1261] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234 [term 1 FOLLOWER]: Refusing update from remote peer 8c1bb58fa8f947f1b2ec7718f60a5600: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:30.212019 2046 consensus_queue.cc:1035] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:30.212144 2022 consensus_queue.cc:1035] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:30.212793 2024 tablet_replica.cc:439] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234: TxnStatusTablet state changed. Reason: New leader 8c1bb58fa8f947f1b2ec7718f60a5600. Latest consensus state: current_term: 1 leader_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } } }
I20260925 13:17:30.212880 2024 tablet_replica.cc:442] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:30.212846 2026 tablet_replica.cc:439] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96: TxnStatusTablet state changed. Reason: New leader 8c1bb58fa8f947f1b2ec7718f60a5600. Latest consensus state: current_term: 1 leader_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } } }
I20260925 13:17:30.212937 2026 tablet_replica.cc:442] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:30.213698 2046 tablet_replica.cc:439] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } } }
I20260925 13:17:30.213793 2046 tablet_replica.cc:442] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:17:30.213966 2022 tablet_replica.cc:439] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } } }
I20260925 13:17:30.214236 2026 tablet_replica.cc:439] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } } }
I20260925 13:17:30.214311 2026 tablet_replica.cc:442] T 8390c6918483471c8727f8ee73d006f0 P 6599055bc62a4825a72cc38641bf0a96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:30.214622 2064 mvcc.cc:205] Tried to move back new op lower bound from 7333241856863854592 to 7333241856832499712. Current Snapshot: MvccSnapshot[applied={T|T < 7333241856863854592}]
I20260925 13:17:30.214607 2024 tablet_replica.cc:439] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1bb58fa8f947f1b2ec7718f60a5600" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38825 } } peers { permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 } } peers { permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 } } }
I20260925 13:17:30.214795 2024 tablet_replica.cc:442] T 8390c6918483471c8727f8ee73d006f0 P a66ecc6d04e74e739b72776e786ad234: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:30.214140 2022 tablet_replica.cc:442] T 8390c6918483471c8727f8ee73d006f0 P 8c1bb58fa8f947f1b2ec7718f60a5600: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:17:30.221819 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:30.219669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.0.98.1:38825)
0925 13:17:30.219726 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:30.219728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:30.219734 (+ 6us) server_negotiation.cc:407] Connection header received
0925 13:17:30.219754 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:30.219755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:30.219763 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:30.219808 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:30.219931 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:30.220439 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:30.221159 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:30.221354 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:30.221408 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:30.221560 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:30.221606 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:30.221648 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:30.221718 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 13:17:30.221740 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 13:17:30.226658 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:30.224345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46921 (local address 127.0.98.1:38825)
0925 13:17:30.224388 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:30.224390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:30.224515 (+ 125us) server_negotiation.cc:407] Connection header received
0925 13:17:30.224598 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:30.224598 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:30.224607 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:30.224656 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:30.224785 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:30.225253 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:30.226051 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:30.226229 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:30.226281 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:30.226427 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:30.226456 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:30.226496 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:30.226556 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 13:17:30.226574 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:17:30.226730 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:30.224280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38825 (local address 127.0.98.1:46921)
0925 13:17:30.224451 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:30.224460 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:30.224516 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:30.224659 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:30.224660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:30.224662 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:17:30.224734 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:30.224736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:30.225319 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:30.225319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:30.225978 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:30.225980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:30.226282 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:30.226283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:30.226350 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:30.226364 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:17:30.226499 (+ 135us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:17:30.226501 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:17:30.226552 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 13:17:30.226589 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260925 13:17:30.228909 1938 raft_consensus.cc:1261] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96 [term 1 FOLLOWER]: Refusing update from remote peer 8c1bb58fa8f947f1b2ec7718f60a5600: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:30.228935 1806 raft_consensus.cc:1261] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234 [term 1 FOLLOWER]: Refusing update from remote peer 8c1bb58fa8f947f1b2ec7718f60a5600: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:30.229120 2075 consensus_queue.cc:1035] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6599055bc62a4825a72cc38641bf0a96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:30.229266 2046 consensus_queue.cc:1035] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66ecc6d04e74e739b72776e786ad234" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:30.248204 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:30.248332 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:30.248984 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03112d80310c497792cc5b871ae0ccc4: Sending DeleteTablet for 3 replicas of tablet 177e6a084a284a1faef4f8ec4d3b30ec
I20260925 13:17:30.249238 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 177e6a084a284a1faef4f8ec4d3b30ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41534
I20260925 13:17:30.249269 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 177e6a084a284a1faef4f8ec4d3b30ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49298
I20260925 13:17:30.249313 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 177e6a084a284a1faef4f8ec4d3b30ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50100
I20260925 13:17:30.249621 2081 tablet_replica.cc:330] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96: stopping tablet replica
I20260925 13:17:30.249713 2079 tablet_replica.cc:330] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600: stopping tablet replica
I20260925 13:17:30.249722 2081 raft_consensus.cc:2229] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:30.249779 2079 raft_consensus.cc:2229] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:17:30.249854 2081 raft_consensus.cc:2258] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:30.249879 2079 raft_consensus.cc:2258] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:30.250198 2081 ts_tablet_manager.cc:1905] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:30.250254 2079 ts_tablet_manager.cc:1905] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:30.251778 2081 ts_tablet_manager.cc:1918] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:17:30.251837 2081 log.cc:1196] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/177e6a084a284a1faef4f8ec4d3b30ec
I20260925 13:17:30.252032 2081 ts_tablet_manager.cc:1939] T 177e6a084a284a1faef4f8ec4d3b30ec P 6599055bc62a4825a72cc38641bf0a96: Deleting consensus metadata
I20260925 13:17:30.252035 2079 ts_tablet_manager.cc:1918] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:17:30.252084 2079 log.cc:1196] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-0/wal/wals/177e6a084a284a1faef4f8ec4d3b30ec
I20260925 13:17:30.252329 1493 catalog_manager.cc:4841] TS 6599055bc62a4825a72cc38641bf0a96 (127.0.98.3:33829): tablet 177e6a084a284a1faef4f8ec4d3b30ec (table test-table [id=e65c1dda42574253b9e72a6e2da5202d]) successfully deleted
I20260925 13:17:30.252432 2079 ts_tablet_manager.cc:1939] T 177e6a084a284a1faef4f8ec4d3b30ec P 8c1bb58fa8f947f1b2ec7718f60a5600: Deleting consensus metadata
I20260925 13:17:30.252702 2080 tablet_replica.cc:330] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234: stopping tablet replica
I20260925 13:17:30.252712 1491 catalog_manager.cc:4841] TS 8c1bb58fa8f947f1b2ec7718f60a5600 (127.0.98.1:38825): tablet 177e6a084a284a1faef4f8ec4d3b30ec (table test-table [id=e65c1dda42574253b9e72a6e2da5202d]) successfully deleted
I20260925 13:17:30.252770 2080 raft_consensus.cc:2229] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:30.252818 2080 raft_consensus.cc:2258] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:30.253041 2080 ts_tablet_manager.cc:1905] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:30.254694 2080 ts_tablet_manager.cc:1918] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:17:30.254760 2080 log.cc:1196] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/177e6a084a284a1faef4f8ec4d3b30ec
I20260925 13:17:30.254948 2080 ts_tablet_manager.cc:1939] T 177e6a084a284a1faef4f8ec4d3b30ec P a66ecc6d04e74e739b72776e786ad234: Deleting consensus metadata
I20260925 13:17:30.255300 1492 catalog_manager.cc:4841] TS a66ecc6d04e74e739b72776e786ad234 (127.0.98.2:43455): tablet 177e6a084a284a1faef4f8ec4d3b30ec (table test-table [id=e65c1dda42574253b9e72a6e2da5202d]) successfully deleted
Sep 25 13:17:30 dist-test-slave-t8d9 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342249, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:17:30.261835 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:30.251735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41546 (local address 127.0.98.1:38825)
0925 13:17:30.251780 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:30.251782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:30.253737 (+ 1955us) server_negotiation.cc:407] Connection header received
0925 13:17:30.253798 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:30.253799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:30.253809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:30.253855 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:30.254393 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:30.254913 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:30.255736 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:30.255928 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:30.255981 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:30.258986 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:30.258997 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:30.258997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:30.259005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:30.260584 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:30.260978 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:30.260981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:30.260981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:30.261034 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:30.261209 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:30.261210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:30.261210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:30.261356 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:30.261462 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:30.261649 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 13:17:30.261728 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 13:17:30.262113 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41546
I20260925 13:17:30.269121 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:30.263033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.0.98.62:33549)
0925 13:17:30.263076 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:30.263078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:30.263139 (+ 61us) server_negotiation.cc:407] Connection header received
0925 13:17:30.263186 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:30.263187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:30.263198 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:30.263276 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:30.263684 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:30.264463 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:30.265146 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:30.265446 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:30.265517 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:30.266770 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:30.266772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:30.266772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:30.266780 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:30.268113 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:30.268417 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:30.268417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:30.268418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:30.268469 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:30.268640 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:30.268641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:30.268641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:30.268748 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:30.268832 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:30.268948 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 13:17:30.269025 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 13:17:30.269361 1505 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60626
I20260925 13:17:30.270035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 1579
I20260925 13:17:30.275074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 1720
I20260925 13:17:30.279901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 1861
I20260925 13:17:30.284744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 1475
2026-09-25T13:17:30Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6804 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:17:30 dist-test-slave-t8d9 krb5kdc[2088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:17:30 dist-test-slave-t8d9 krb5kdc[2088](info): set up 2 sockets
Sep 25 13:17:30 dist-test-slave-t8d9 krb5kdc[2088](info): commencing operation
krb5kdc: starting...
W20260925 13:17:31.237039 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:17:31 dist-test-slave-t8d9 krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:17:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:17:31.349881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42007
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:31.428701 2101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:31.428928 2101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:31.428951 2101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:31.430213 2101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:31.430262 2101 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 13:17:31.430274 2101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:31.430286 2101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:31.430298 2101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:31.430310 2101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:31.432013 2101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42007
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:31.432277 2101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:31.432502 2101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:31.437126 2107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:31.437120 2106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:31.437115 2109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:31.437201 2101 server_base.cc:1034] running on GCE node
I20260925 13:17:31.437588 2101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:31.437865 2101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:31.439023 2101 hybrid_clock.cc:648] HybridClock initialized: now 1790342251439009 us; error 30 us; skew 500 ppm
Sep 25 13:17:31 dist-test-slave-t8d9 krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:31.441439 2101 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:17:31.441883 2101 webserver.cc:460] Webserver started at http://127.0.98.62:34169/ using document root <none> and password file <none>
I20260925 13:17:31.442096 2101 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:31.442142 2101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:31.442268 2101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:31.443323 2101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "2fbe722714724aa09164062b7d2002b7"
format_stamp: "Formatted at 2026-09-25 13:17:31 on dist-test-slave-t8d9"
server_key: "acab3950aadf1cb627c3f1a3145c7e74"
server_key_iv: "44db8061e693a48033d00ad667b3715d"
server_key_version: "encryptionkey@0"
I20260925 13:17:31.443658 2101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "2fbe722714724aa09164062b7d2002b7"
format_stamp: "Formatted at 2026-09-25 13:17:31 on dist-test-slave-t8d9"
server_key: "acab3950aadf1cb627c3f1a3145c7e74"
server_key_iv: "44db8061e693a48033d00ad667b3715d"
server_key_version: "encryptionkey@0"
I20260925 13:17:31.445004 2101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:17:31.445741 2116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:31.445894 2101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:17:31.445982 2101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "2fbe722714724aa09164062b7d2002b7"
format_stamp: "Formatted at 2026-09-25 13:17:31 on dist-test-slave-t8d9"
server_key: "acab3950aadf1cb627c3f1a3145c7e74"
server_key_iv: "44db8061e693a48033d00ad667b3715d"
server_key_version: "encryptionkey@0"
I20260925 13:17:31.446046 2101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:31.463001 2101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:31.465112 2101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:31.465257 2101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:31.469691 2101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42007
I20260925 13:17:31.469750 2178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42007 every 8 connection(s)
I20260925 13:17:31.470079 2101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:31.470706 2179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:31.473001 2179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7: Bootstrap starting.
I20260925 13:17:31.473634 2179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:31.473901 2179 log.cc:824] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:31.474560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 2101
I20260925 13:17:31.474663 2179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7: No bootstrap required, opened a new log
I20260925 13:17:31.474684 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:31.474849 392 external_mini_cluster.cc:1428] Setting key 8681137a80f5369c0de9db893e76545e
I20260925 13:17:31.476941 2179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbe722714724aa09164062b7d2002b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42007 } }
I20260925 13:17:31.477084 2179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:31.477125 2179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbe722714724aa09164062b7d2002b7, State: Initialized, Role: FOLLOWER
I20260925 13:17:31.477273 2179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbe722714724aa09164062b7d2002b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42007 } }
I20260925 13:17:31.477341 2179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:31.477371 2179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:31.477418 2179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:31.477989 2179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbe722714724aa09164062b7d2002b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42007 } }
I20260925 13:17:31.478080 2179 leader_election.cc:302] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fbe722714724aa09164062b7d2002b7; no voters:
I20260925 13:17:31.478250 2179 leader_election.cc:288] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:31.478279 2184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:31.478386 2184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [term 1 LEADER]: Becoming Leader. State: Replica: 2fbe722714724aa09164062b7d2002b7, State: Running, Role: LEADER
I20260925 13:17:31.478466 2184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbe722714724aa09164062b7d2002b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42007 } }
I20260925 13:17:31.478509 2179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:31.478834 2186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fbe722714724aa09164062b7d2002b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbe722714724aa09164062b7d2002b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42007 } } }
I20260925 13:17:31.478910 2186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:31.479045 2185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fbe722714724aa09164062b7d2002b7. Latest consensus state: current_term: 1 leader_uuid: "2fbe722714724aa09164062b7d2002b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbe722714724aa09164062b7d2002b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42007 } } }
I20260925 13:17:31.479103 2185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:31.479215 2190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:31.479697 2190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:31.481338 2190 catalog_manager.cc:1347] Generated new cluster ID: 2c6bef1a900f42f98a93491f2775ede2
I20260925 13:17:31.481392 2190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:17:31 dist-test-slave-t8d9 krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:31.487084 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.475393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.0.98.62:42007)
0925 13:17:31.475596 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:31.475603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:31.475628 (+ 25us) server_negotiation.cc:407] Connection header received
0925 13:17:31.476926 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:31.476934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:31.476991 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:31.477134 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:31.477854 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.479129 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.480253 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.480732 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.480874 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:31.483642 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:31.483646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:31.483647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:31.483659 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:31.485505 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.485913 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.485915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.485916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.486020 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.486211 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.486212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.486212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.486498 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:31.486633 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:31.486754 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 13:17:31.486885 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":89,"threads_started":1}
I20260925 13:17:31.490068 2190 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:31.490548 2190 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:31.506645 2190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7: Generated new TSK 0
I20260925 13:17:31.506909 2190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:17:31.536981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42007
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:31.617188 2205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:31.617404 2205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:31.617430 2205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:31.618584 2205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:31.618633 2205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:31.618651 2205 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 13:17:31.618662 2205 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:17:31.618677 2205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:17:31.620103 2205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:31.620307 2205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:31.620496 2205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:31.625231 2210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:31.625283 2213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:31.625239 2211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:31.625515 2205 server_base.cc:1034] running on GCE node
I20260925 13:17:31.625789 2205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:31.626013 2205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:31.627154 2205 hybrid_clock.cc:648] HybridClock initialized: now 1790342251627138 us; error 27 us; skew 500 ppm
Sep 25 13:17:31 dist-test-slave-t8d9 krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:31.629647 2205 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:17:31.630112 2205 webserver.cc:460] Webserver started at http://127.0.98.1:40727/ using document root <none> and password file <none>
I20260925 13:17:31.630326 2205 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:31.630373 2205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:31.630501 2205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:31.631598 2205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "cec0287e6b3344e0997af29d050222b1"
format_stamp: "Formatted at 2026-09-25 13:17:31 on dist-test-slave-t8d9"
server_key: "7c656f00c3be379e1b552ada3d81d470"
server_key_iv: "1cc20ae0ed3f85859a700d91582d931d"
server_key_version: "encryptionkey@0"
I20260925 13:17:31.631943 2205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cec0287e6b3344e0997af29d050222b1"
format_stamp: "Formatted at 2026-09-25 13:17:31 on dist-test-slave-t8d9"
server_key: "7c656f00c3be379e1b552ada3d81d470"
server_key_iv: "1cc20ae0ed3f85859a700d91582d931d"
server_key_version: "encryptionkey@0"
I20260925 13:17:31.633203 2205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:17:31.633955 2220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:31.634104 2205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:31.634183 2205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "cec0287e6b3344e0997af29d050222b1"
format_stamp: "Formatted at 2026-09-25 13:17:31 on dist-test-slave-t8d9"
server_key: "7c656f00c3be379e1b552ada3d81d470"
server_key_iv: "1cc20ae0ed3f85859a700d91582d931d"
server_key_version: "encryptionkey@0"
I20260925 13:17:31.634243 2205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:31.645752 2205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:31.647468 2205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:31.647607 2205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:31.648121 2205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:31.648168 2205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:31.648198 2205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:31.648216 2205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 13:17:31 dist-test-slave-t8d9 krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:31.654881 2205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39685
I20260925 13:17:31.655236 2205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:17:31.655282 2335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39685 every 8 connection(s)
I20260925 13:17:31.657266 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.648522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35979 (local address 127.0.98.62:42007)
0925 13:17:31.648568 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:31.648570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:31.648797 (+ 227us) server_negotiation.cc:407] Connection header received
0925 13:17:31.649307 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:31.649308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:31.649321 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:31.649371 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:31.650126 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.650641 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.651707 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.652024 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.652097 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:31.654423 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:31.654425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:31.654425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:31.654437 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:31.655977 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.656349 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.656351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.656351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.656413 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.656631 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.656632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.656632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.656762 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:31.656819 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:31.657078 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 13:17:31.657172 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:17:31.657428 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.648528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42007 (local address 127.0.98.1:35979)
0925 13:17:31.648683 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:31.648717 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:31.649210 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:31.649444 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:31.649448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:31.649830 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:31.650031 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:31.650037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.650726 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.650727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:31.651630 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:31.651633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.652131 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.652132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:31.652289 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:31.652860 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:31.652871 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:31.654342 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:31.656059 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:31.656064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:31.656067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:31.656283 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:31.656465 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:31.656465 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:31.656466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:31.656574 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:31.656815 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:31.656821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:31.656958 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 13:17:31.657047 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 13:17:31.657204 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260925 13:17:31.657923 2336 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42007
I20260925 13:17:31.658002 2336 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:31.658177 2336 heartbeater.cc:511] Master 127.0.98.62:42007 requested a full tablet report, sending...
I20260925 13:17:31.658699 2132 ts_manager.cc:194] Registered new tserver with Master: cec0287e6b3344e0997af29d050222b1 (127.0.98.1:39685)
I20260925 13:17:31.659495 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35979
I20260925 13:17:31.661621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 2205
I20260925 13:17:31.661749 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:17:31.661949 392 external_mini_cluster.cc:1428] Setting key 564f452ae9941db4317f00f017abfe5a
I20260925 13:17:31.665004 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.658843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42007 (local address 127.0.98.1:58889)
0925 13:17:31.658977 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:31.658989 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:31.659056 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:31.659810 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:31.659811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:31.659996 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:31.660074 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:31.660077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.660695 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.660696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:31.661243 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:31.661245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.661661 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.661662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:31.661749 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:31.662273 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:31.662277 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:31.662584 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:31.663972 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:31.663973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:31.663974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:31.664185 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:31.664353 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:31.664353 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:31.664354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:31.664413 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:31.664731 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:31.664732 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:31.664773 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 13:17:31.664833 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 13:17:31.664926 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260925 13:17:31.665081 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.658923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58889 (local address 127.0.98.62:42007)
0925 13:17:31.659695 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:31.659696 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:31.659703 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:17:31.659720 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:31.659721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:31.659734 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:31.659801 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:31.660145 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.660598 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.661307 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.661519 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.661607 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:31.662653 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:31.662654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:31.662655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:31.662661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:31.663892 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.664253 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.664254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.664254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.664302 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.664493 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.664495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.664495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.664665 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:31.664729 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:31.664886 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 13:17:31.664981 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:17:31.686399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42007
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:31.768074 2345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:31.768309 2345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:31.768344 2345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:31.769577 2345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:31.769624 2345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:31.769654 2345 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 13:17:31.769670 2345 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:17:31.769701 2345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:17:31.771250 2345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:31.771499 2345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:31.771729 2345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:31.776466 2353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:31.776582 2345 server_base.cc:1034] running on GCE node
W20260925 13:17:31.776674 2351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:31.776487 2350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:31.776930 2345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:31.777184 2345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:31.778317 2345 hybrid_clock.cc:648] HybridClock initialized: now 1790342251778300 us; error 35 us; skew 500 ppm
Sep 25 13:17:31 dist-test-slave-t8d9 krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:31.780751 2345 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:17:31.781158 2345 webserver.cc:460] Webserver started at http://127.0.98.2:43319/ using document root <none> and password file <none>
I20260925 13:17:31.781354 2345 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:31.781396 2345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:31.781483 2345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:31.782527 2345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "fdbfc877556b4e598b6add2316605c8e"
format_stamp: "Formatted at 2026-09-25 13:17:31 on dist-test-slave-t8d9"
server_key: "4d01aabf58f7607c7a5aa78ecfb707a0"
server_key_iv: "c7ecb398984bcab4ba91f6d848bea938"
server_key_version: "encryptionkey@0"
I20260925 13:17:31.782820 2345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fdbfc877556b4e598b6add2316605c8e"
format_stamp: "Formatted at 2026-09-25 13:17:31 on dist-test-slave-t8d9"
server_key: "4d01aabf58f7607c7a5aa78ecfb707a0"
server_key_iv: "c7ecb398984bcab4ba91f6d848bea938"
server_key_version: "encryptionkey@0"
I20260925 13:17:31.784042 2345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:17:31.784703 2360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:31.784875 2345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:31.784946 2345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "fdbfc877556b4e598b6add2316605c8e"
format_stamp: "Formatted at 2026-09-25 13:17:31 on dist-test-slave-t8d9"
server_key: "4d01aabf58f7607c7a5aa78ecfb707a0"
server_key_iv: "c7ecb398984bcab4ba91f6d848bea938"
server_key_version: "encryptionkey@0"
I20260925 13:17:31.785014 2345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:31.803210 2345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:31.805003 2345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:31.805130 2345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:31.805677 2345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:31.805718 2345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:31.805740 2345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:31.805751 2345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:31.811726 2345 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44693
I20260925 13:17:31.811796 2475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44693 every 8 connection(s)
I20260925 13:17:31.812129 2345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:17:31 dist-test-slave-t8d9 krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:31.815111 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.806098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36793 (local address 127.0.98.62:42007)
0925 13:17:31.806159 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:31.806161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:31.806376 (+ 215us) server_negotiation.cc:407] Connection header received
0925 13:17:31.806876 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:31.806877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:31.806890 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:31.806967 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:31.807786 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.808525 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.810038 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.810222 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.810275 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:31.812696 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:31.812698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:31.812698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:31.812706 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:31.813805 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.814204 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.814206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.814206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.814275 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.814490 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.814491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.814492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.814626 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:31.814685 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:31.814940 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 13:17:31.815022 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:17:31.815268 2369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.806078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42007 (local address 127.0.98.2:36793)
0925 13:17:31.806242 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:31.806282 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:31.806780 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:31.807040 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:31.807044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:31.807389 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:31.807680 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:31.807687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.808616 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.808617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:31.809941 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:31.809944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.810298 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.810300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:31.810454 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:31.811063 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:31.811071 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:31.812583 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:31.813869 (+ 1286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:31.813875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:31.813876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:31.814127 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:31.814330 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:31.814331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:31.814331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:31.814437 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:31.814675 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:31.814681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:31.814822 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 13:17:31.814909 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 13:17:31.815062 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":103,"threads_started":1}
I20260925 13:17:31.815676 2476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42007
I20260925 13:17:31.815752 2476 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:31.815925 2476 heartbeater.cc:511] Master 127.0.98.62:42007 requested a full tablet report, sending...
I20260925 13:17:31.816295 2133 ts_manager.cc:194] Registered new tserver with Master: fdbfc877556b4e598b6add2316605c8e (127.0.98.2:44693)
I20260925 13:17:31.816776 2133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36793
I20260925 13:17:31.821099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 2345
I20260925 13:17:31.821187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:17:31.821342 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.816713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48289 (local address 127.0.98.62:42007)
0925 13:17:31.816764 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:31.816766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:31.816805 (+ 39us) server_negotiation.cc:407] Connection header received
0925 13:17:31.816863 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:31.816863 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:31.816872 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:31.816927 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:31.817254 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.817834 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.818479 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.818643 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.818682 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:31.819387 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:31.819388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:31.819388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:31.819394 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:31.820417 (+ 1023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.820699 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.820700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.820700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.820740 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.820869 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.820870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.820870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.821029 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:31.821098 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:31.821192 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 13:17:31.821261 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:17:31.821343 2482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.816615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42007 (local address 127.0.98.2:48289)
0925 13:17:31.816747 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:31.816758 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:31.816819 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:31.816931 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:31.816931 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:31.817123 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:31.817187 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:31.817189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.817931 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.817932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:31.818421 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:31.818422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.818687 (+ 265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.818688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:31.818738 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:31.819070 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:31.819074 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:31.819322 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:31.820476 (+ 1154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:31.820477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:31.820477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:31.820647 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:31.820791 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:31.820791 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:31.820791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:31.820832 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:31.821085 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:31.821086 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:31.821116 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 13:17:31.821167 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 13:17:31.821261 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260925 13:17:31.821390 392 external_mini_cluster.cc:1428] Setting key 672b809572dd4a5650708da4e59d2d8a
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:17:31.844264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42007
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:31.923631 2485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:31.923849 2485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:31.923880 2485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:31.925056 2485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:31.925098 2485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:31.925133 2485 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 13:17:31.925149 2485 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:17:31.925173 2485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:17:31.926694 2485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:31.926949 2485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:31.927166 2485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:31.932030 2490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:31.932106 2485 server_base.cc:1034] running on GCE node
W20260925 13:17:31.932015 2491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:31.932016 2493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:31.932433 2485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:31.932684 2485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:31.933836 2485 hybrid_clock.cc:648] HybridClock initialized: now 1790342251933819 us; error 32 us; skew 500 ppm
Sep 25 13:17:31 dist-test-slave-t8d9 krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:31.936179 2485 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:17:31.936609 2485 webserver.cc:460] Webserver started at http://127.0.98.3:44525/ using document root <none> and password file <none>
I20260925 13:17:31.936846 2485 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:31.936894 2485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:31.937038 2485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:31.938113 2485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "d623e6a106a54352b3d1510ad55859b1"
format_stamp: "Formatted at 2026-09-25 13:17:31 on dist-test-slave-t8d9"
server_key: "299c39ba27082ad5fd7f53cf988890a2"
server_key_iv: "f69ca7453eb54c92fccb76f37ff77268"
server_key_version: "encryptionkey@0"
I20260925 13:17:31.938444 2485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d623e6a106a54352b3d1510ad55859b1"
format_stamp: "Formatted at 2026-09-25 13:17:31 on dist-test-slave-t8d9"
server_key: "299c39ba27082ad5fd7f53cf988890a2"
server_key_iv: "f69ca7453eb54c92fccb76f37ff77268"
server_key_version: "encryptionkey@0"
I20260925 13:17:31.939697 2485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:17:31.940435 2500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:31.940609 2485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:31.940694 2485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "d623e6a106a54352b3d1510ad55859b1"
format_stamp: "Formatted at 2026-09-25 13:17:31 on dist-test-slave-t8d9"
server_key: "299c39ba27082ad5fd7f53cf988890a2"
server_key_iv: "f69ca7453eb54c92fccb76f37ff77268"
server_key_version: "encryptionkey@0"
I20260925 13:17:31.940778 2485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:31.955523 2485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:31.957348 2485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:31.957472 2485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:31.958032 2485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:31.958072 2485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:31.958094 2485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:31.958118 2485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:31.964630 2485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45041
I20260925 13:17:31.964723 2615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45041 every 8 connection(s)
Sep 25 13:17:31 dist-test-slave-t8d9 krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:31.965013 2485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:17:31.968467 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.958441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41447 (local address 127.0.98.62:42007)
0925 13:17:31.958502 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:31.958504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:31.958719 (+ 215us) server_negotiation.cc:407] Connection header received
0925 13:17:31.959298 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:31.959300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:31.959314 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:31.959362 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:31.960206 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.960954 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.962322 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.962610 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.962682 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:31.965273 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:31.965275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:31.965275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:31.965284 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:31.966974 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.967416 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.967417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.967417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.967480 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.967739 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.967740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.967741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.967872 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:31.967936 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:31.968279 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 13:17:31.968372 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:17:31.968638 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.958418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42007 (local address 127.0.98.3:41447)
0925 13:17:31.958591 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:31.958632 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:31.959176 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:31.959425 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:31.959429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:31.959784 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:31.960099 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:31.960107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.961047 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.961048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:31.962228 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:31.962231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.962704 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.962705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:31.962874 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:31.963443 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:31.963454 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:31.965169 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:31.967076 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:31.967086 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:31.967088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:31.967334 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:31.967544 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:31.967545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:31.967546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:31.967679 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:31.967932 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:31.967939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:31.968111 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 13:17:31.968232 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 13:17:31.968409 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":109,"threads_started":1}
I20260925 13:17:31.969080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 2485
I20260925 13:17:31.969177 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:17:31.969224 2616 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42007
I20260925 13:17:31.969326 2616 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:31.969411 392 external_mini_cluster.cc:1428] Setting key 03b613900d2200ffd75579e5b2a2ba88
I20260925 13:17:31.969518 2616 heartbeater.cc:511] Master 127.0.98.62:42007 requested a full tablet report, sending...
I20260925 13:17:31.969882 2132 ts_manager.cc:194] Registered new tserver with Master: d623e6a106a54352b3d1510ad55859b1 (127.0.98.3:45041)
I20260925 13:17:31.970299 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41447
I20260925 13:17:31.970777 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:17:31.976684 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.970355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43029 (local address 127.0.98.62:42007)
0925 13:17:31.970491 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:31.970493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:31.970740 (+ 247us) server_negotiation.cc:407] Connection header received
0925 13:17:31.970806 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:31.970807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:31.970818 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:31.970866 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:31.971295 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.971754 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.972657 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.972862 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.972922 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:31.973975 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:31.973976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:31.973977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:31.973984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:31.975549 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.975896 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.975897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.975897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.975962 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.976136 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.976137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.976137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.976271 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:31.976334 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:31.976535 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 13:17:31.976616 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260925 13:17:31.976729 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.970416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42007 (local address 127.0.98.3:43029)
0925 13:17:31.970679 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:31.970690 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:31.970752 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:31.970913 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:31.970914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:31.971132 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:31.971207 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:31.971209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.971852 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.971853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:31.972581 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:31.972584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.972925 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.972926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:31.973003 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:31.973510 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:31.973515 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:31.973881 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:31.975652 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:31.975653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:31.975653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:31.975825 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:31.976026 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:31.976026 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:31.976026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:31.976084 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:31.976332 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:31.976333 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:31.976393 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 13:17:31.976497 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 13:17:31.976621 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":81,"threads_started":1}
Sep 25 13:17:31 dist-test-slave-t8d9 krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 13:17:31 dist-test-slave-t8d9 krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:31.993340 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.986726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.0.98.62:42007)
0925 13:17:31.986781 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:31.986783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:31.986816 (+ 33us) server_negotiation.cc:407] Connection header received
0925 13:17:31.986837 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:31.986838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:31.986847 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:31.986885 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:31.987475 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.988009 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.988747 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:31.988929 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:31.988979 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:31.991164 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:31.991165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:31.991166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:31.991173 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:31.992345 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.992663 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.992664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.992664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.992714 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:31.992876 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:31.992877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:31.992877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:31.992977 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:31.993061 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:31.993171 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 13:17:31.993250 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:17:31.994825 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:17:31.996390 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:32.002658 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.999012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43756 (local address 127.0.98.1:39685)
0925 13:17:31.999279 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:31.999283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:31.999292 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:31.999325 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:31.999327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:31.999342 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:31.999406 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:31.999578 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.000637 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.001676 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.002244 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.002307 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.002371 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:32.002501 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 13:17:32.002540 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":89,"threads_started":1}
I20260925 13:17:32.002655 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.998551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44693 (local address 127.0.0.1:49068)
0925 13:17:31.998668 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:31.998711 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:31.998775 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:31.999211 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:31.999215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:31.999218 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:31.999296 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:31.999299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.000249 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.000250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.001432 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.001435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.002362 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.002363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.002438 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.002443 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:17:32.002471 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 13:17:32.002554 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20260925 13:17:32.002940 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.998749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45041 (local address 127.0.0.1:52622)
0925 13:17:31.998987 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:31.998996 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:31.999037 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:31.999272 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:31.999273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:31.999275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:31.999330 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:31.999332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.000640 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.000641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.001787 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.001790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.002738 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.002739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.002795 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.002799 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:32.002842 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:17:32.002880 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":9,"thread_start_us":73,"threads_started":1}
I20260925 13:17:32.002661 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.998905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39685 (local address 127.0.0.1:43756)
0925 13:17:31.999104 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:31.999112 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:31.999145 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:31.999459 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:31.999460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:31.999462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:31.999498 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:31.999499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.000696 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.000697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.001594 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.001597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.002320 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.002321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.002401 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.002415 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 13:17:32.002481 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 13:17:32.002554 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":27,"thread_start_us":41,"threads_started":1}
I20260925 13:17:32.003116 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.998664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.0.98.2:44693)
0925 13:17:31.999017 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:31.999022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:31.999032 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:17:31.999075 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:31.999077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:31.999093 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:31.999376 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:31.999439 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.000172 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.001517 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.002261 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.002878 (+ 617us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.002950 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:32.002977 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 13:17:32.003020 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":80,"threads_started":1}
I20260925 13:17:32.003304 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:31.998838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.0.98.3:45041)
0925 13:17:31.999009 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:31.999014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:31.999026 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:17:31.999101 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:31.999103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:31.999119 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:31.999193 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:31.999402 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.000539 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.001867 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.002589 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.002651 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.002705 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:32.003152 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 13:17:32.003197 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260925 13:17:32.003731 2270 tablet_service.cc:1467] Processing CreateTablet for tablet 143b7457b80848f68470b514f0742579 (DEFAULT_TABLE table=test-table [id=c8fe432b8ee146e29392d3ea1ca2e9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:32.003715 2410 tablet_service.cc:1467] Processing CreateTablet for tablet 143b7457b80848f68470b514f0742579 (DEFAULT_TABLE table=test-table [id=c8fe432b8ee146e29392d3ea1ca2e9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:32.003837 2550 tablet_service.cc:1467] Processing CreateTablet for tablet 143b7457b80848f68470b514f0742579 (DEFAULT_TABLE table=test-table [id=c8fe432b8ee146e29392d3ea1ca2e9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:32.004081 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143b7457b80848f68470b514f0742579. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:32.004088 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143b7457b80848f68470b514f0742579. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:32.004083 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143b7457b80848f68470b514f0742579. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:32.007040 2636 tablet_bootstrap.cc:492] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e: Bootstrap starting.
I20260925 13:17:32.007360 2637 tablet_bootstrap.cc:492] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1: Bootstrap starting.
I20260925 13:17:32.007519 2638 tablet_bootstrap.cc:492] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1: Bootstrap starting.
I20260925 13:17:32.007879 2636 tablet_bootstrap.cc:654] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:32.008095 2638 tablet_bootstrap.cc:654] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:32.008163 2637 tablet_bootstrap.cc:654] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:32.008195 2636 log.cc:824] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:32.008400 2638 log.cc:824] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:32.008450 2637 log.cc:824] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:32.009037 2637 tablet_bootstrap.cc:492] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1: No bootstrap required, opened a new log
I20260925 13:17:32.009037 2636 tablet_bootstrap.cc:492] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e: No bootstrap required, opened a new log
I20260925 13:17:32.009053 2638 tablet_bootstrap.cc:492] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1: No bootstrap required, opened a new log
I20260925 13:17:32.009104 2638 ts_tablet_manager.cc:1397] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:17:32.009104 2636 ts_tablet_manager.cc:1397] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:17:32.009104 2637 ts_tablet_manager.cc:1397] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:17:32.010512 2638 raft_consensus.cc:348] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } }
I20260925 13:17:32.010648 2638 raft_consensus.cc:374] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:32.010679 2638 raft_consensus.cc:729] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0287e6b3344e0997af29d050222b1, State: Initialized, Role: FOLLOWER
I20260925 13:17:32.010794 2638 consensus_queue.cc:260] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } }
I20260925 13:17:32.011065 2638 ts_tablet_manager.cc:1428] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:17:32.011070 2336 heartbeater.cc:503] Master 127.0.98.62:42007 was elected leader, sending a full tablet report...
I20260925 13:17:32.011037 2637 raft_consensus.cc:348] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } }
I20260925 13:17:32.011081 2636 raft_consensus.cc:348] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } }
I20260925 13:17:32.011198 2636 raft_consensus.cc:374] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:32.011196 2637 raft_consensus.cc:374] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:32.011228 2636 raft_consensus.cc:729] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdbfc877556b4e598b6add2316605c8e, State: Initialized, Role: FOLLOWER
I20260925 13:17:32.011228 2637 raft_consensus.cc:729] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d623e6a106a54352b3d1510ad55859b1, State: Initialized, Role: FOLLOWER
I20260925 13:17:32.011332 2636 consensus_queue.cc:260] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } }
I20260925 13:17:32.011332 2637 consensus_queue.cc:260] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } }
I20260925 13:17:32.011579 2637 ts_tablet_manager.cc:1428] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 13:17:32.011579 2636 ts_tablet_manager.cc:1428] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 13:17:32.011615 2616 heartbeater.cc:503] Master 127.0.98.62:42007 was elected leader, sending a full tablet report...
I20260925 13:17:32.011878 2476 heartbeater.cc:503] Master 127.0.98.62:42007 was elected leader, sending a full tablet report...
W20260925 13:17:32.062737 2477 tablet.cc:2369] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:17:32.067214 2643 raft_consensus.cc:482] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:32.067342 2643 raft_consensus.cc:504] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } }
I20260925 13:17:32.067693 2643 leader_election.cc:288] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdbfc877556b4e598b6add2316605c8e (127.0.98.2:44693), cec0287e6b3344e0997af29d050222b1 (127.0.98.1:39685)
I20260925 13:17:32.070631 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.068093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53539 (local address 127.0.98.1:39685)
0925 13:17:32.068133 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:32.068135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:32.068225 (+ 90us) server_negotiation.cc:407] Connection header received
0925 13:17:32.068258 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:32.068259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:32.068272 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:32.068317 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:32.068456 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.068950 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.069943 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.070326 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.070382 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.070394 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:32.070523 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 13:17:32.070555 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:17:32.070647 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.067939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54077 (local address 127.0.98.2:44693)
0925 13:17:32.067991 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:32.067993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:32.068011 (+ 18us) server_negotiation.cc:407] Connection header received
0925 13:17:32.068102 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:32.068103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:32.068115 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:32.068184 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:32.068339 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.068827 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.069928 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.070345 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.070405 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.070415 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:32.070545 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 13:17:32.070581 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:17:32.070662 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.067865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44693 (local address 127.0.98.3:54077)
0925 13:17:32.067960 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:32.067973 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:32.068021 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:32.068208 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:32.068210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:32.068212 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:32.068276 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.068278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.068898 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.068899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.069846 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.069849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.070410 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.070411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.070488 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.070491 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:32.070534 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:17:32.070572 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 13:17:32.070667 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.067973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39685 (local address 127.0.98.3:53539)
0925 13:17:32.068164 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:32.068171 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:32.068213 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:32.068331 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:32.068332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:32.068333 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:32.068403 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.068405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.069019 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.069020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.069862 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.069864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.070398 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.070399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.070462 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.070466 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:32.070509 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:17:32.070548 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260925 13:17:32.071066 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143b7457b80848f68470b514f0742579" candidate_uuid: "d623e6a106a54352b3d1510ad55859b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdbfc877556b4e598b6add2316605c8e" is_pre_election: true
I20260925 13:17:32.071197 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143b7457b80848f68470b514f0742579" candidate_uuid: "d623e6a106a54352b3d1510ad55859b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0287e6b3344e0997af29d050222b1" is_pre_election: true
I20260925 13:17:32.071236 2430 raft_consensus.cc:2454] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d623e6a106a54352b3d1510ad55859b1 in term 0.
I20260925 13:17:32.071324 2290 raft_consensus.cc:2454] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d623e6a106a54352b3d1510ad55859b1 in term 0.
I20260925 13:17:32.071458 2502 leader_election.cc:302] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d623e6a106a54352b3d1510ad55859b1, fdbfc877556b4e598b6add2316605c8e; no voters:
I20260925 13:17:32.071587 2643 raft_consensus.cc:2781] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:32.071643 2643 raft_consensus.cc:482] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:32.071666 2643 raft_consensus.cc:3037] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:32.072317 2643 raft_consensus.cc:504] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } }
I20260925 13:17:32.072422 2643 leader_election.cc:288] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [CANDIDATE]: Term 1 election: Requested vote from peers fdbfc877556b4e598b6add2316605c8e (127.0.98.2:44693), cec0287e6b3344e0997af29d050222b1 (127.0.98.1:39685)
I20260925 13:17:32.072574 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143b7457b80848f68470b514f0742579" candidate_uuid: "d623e6a106a54352b3d1510ad55859b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdbfc877556b4e598b6add2316605c8e"
I20260925 13:17:32.072656 2430 raft_consensus.cc:3037] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:32.072641 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143b7457b80848f68470b514f0742579" candidate_uuid: "d623e6a106a54352b3d1510ad55859b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0287e6b3344e0997af29d050222b1"
I20260925 13:17:32.072722 2290 raft_consensus.cc:3037] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:32.073360 2430 raft_consensus.cc:2454] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d623e6a106a54352b3d1510ad55859b1 in term 1.
I20260925 13:17:32.073465 2290 raft_consensus.cc:2454] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d623e6a106a54352b3d1510ad55859b1 in term 1.
I20260925 13:17:32.073508 2502 leader_election.cc:302] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d623e6a106a54352b3d1510ad55859b1, fdbfc877556b4e598b6add2316605c8e; no voters:
I20260925 13:17:32.073626 2643 raft_consensus.cc:2781] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:32.073793 2643 raft_consensus.cc:686] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [term 1 LEADER]: Becoming Leader. State: Replica: d623e6a106a54352b3d1510ad55859b1, State: Running, Role: LEADER
I20260925 13:17:32.073887 2643 consensus_queue.cc:237] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } }
I20260925 13:17:32.074678 2132 catalog_manager.cc:5495] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d623e6a106a54352b3d1510ad55859b1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d623e6a106a54352b3d1510ad55859b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } health_report { overall_health: UNKNOWN } } }
Sep 25 13:17:32 dist-test-slave-t8d9 krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:32.116709 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.110376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.0.98.62:42007)
0925 13:17:32.110449 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:32.110451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:32.110526 (+ 75us) server_negotiation.cc:407] Connection header received
0925 13:17:32.110630 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:32.110630 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:32.110638 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:32.110684 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:32.111157 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.111633 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.112311 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.112483 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.112535 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.114522 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:32.114523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:32.114523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:32.114530 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:32.115662 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:32.116035 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:32.116036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:32.116036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:32.116085 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:32.116236 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:32.116237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:32.116238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:32.116336 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:32.116389 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:32.116521 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 13:17:32.116610 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 13:17:32.116784 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.110274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42007 (local address 127.0.0.1:40240)
0925 13:17:32.110452 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:32.110462 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:32.110537 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:32.110697 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:32.110698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:32.111017 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:32.111062 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.111064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.111700 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.111701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.112237 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.112239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.112539 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.112540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.112589 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.113096 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:32.113101 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:32.114447 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:32.115737 (+ 1290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:32.115744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:32.115747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:32.115975 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:32.116135 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:32.116136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:32.116136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:32.116190 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:32.116386 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:32.116393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:32.116438 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 13:17:32.116496 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 13:17:32.116676 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":81,"thread_start_us":107,"threads_started":1}
I20260925 13:17:32.118541 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40240:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 13:17:32.118757 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:32.120071 2410 tablet_service.cc:1467] Processing CreateTablet for tablet 7fb69ce475e24c2b8bfeb1e0484dfd64 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c853b08506134e618d6edd76afe89301]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:17:32.120047 2270 tablet_service.cc:1467] Processing CreateTablet for tablet 7fb69ce475e24c2b8bfeb1e0484dfd64 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c853b08506134e618d6edd76afe89301]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:17:32.120087 2550 tablet_service.cc:1467] Processing CreateTablet for tablet 7fb69ce475e24c2b8bfeb1e0484dfd64 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c853b08506134e618d6edd76afe89301]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:17:32.120226 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fb69ce475e24c2b8bfeb1e0484dfd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:32.120226 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fb69ce475e24c2b8bfeb1e0484dfd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:32.120229 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fb69ce475e24c2b8bfeb1e0484dfd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:32.121685 2637 tablet_bootstrap.cc:492] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1: Bootstrap starting.
I20260925 13:17:32.122087 2637 tablet_bootstrap.cc:654] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:32.122392 2636 tablet_bootstrap.cc:492] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e: Bootstrap starting.
I20260925 13:17:32.122736 2637 tablet_bootstrap.cc:492] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1: No bootstrap required, opened a new log
I20260925 13:17:32.122790 2637 ts_tablet_manager.cc:1397] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:32.122946 2637 raft_consensus.cc:348] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.123021 2637 raft_consensus.cc:374] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:32.123023 2636 tablet_bootstrap.cc:654] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:32.123049 2637 raft_consensus.cc:729] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d623e6a106a54352b3d1510ad55859b1, State: Initialized, Role: FOLLOWER
I20260925 13:17:32.123103 2637 consensus_queue.cc:260] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.123217 2637 ts_tablet_manager.cc:1428] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:32.123257 2643 tablet_replica.cc:439] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } }
I20260925 13:17:32.123368 2643 tablet_replica.cc:442] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:32.123395 2638 tablet_bootstrap.cc:492] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1: Bootstrap starting.
I20260925 13:17:32.123725 2636 tablet_bootstrap.cc:492] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e: No bootstrap required, opened a new log
I20260925 13:17:32.123790 2636 ts_tablet_manager.cc:1397] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:32.123934 2636 raft_consensus.cc:348] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.124001 2636 raft_consensus.cc:374] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:32.124020 2636 raft_consensus.cc:729] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdbfc877556b4e598b6add2316605c8e, State: Initialized, Role: FOLLOWER
I20260925 13:17:32.124027 2638 tablet_bootstrap.cc:654] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:32.124061 2636 consensus_queue.cc:260] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.124140 2636 ts_tablet_manager.cc:1428] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:32.124290 2644 tablet_replica.cc:439] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } }
I20260925 13:17:32.124372 2644 tablet_replica.cc:442] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:32.124665 2638 tablet_bootstrap.cc:492] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1: No bootstrap required, opened a new log
I20260925 13:17:32.124713 2638 ts_tablet_manager.cc:1397] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:32.124845 2638 raft_consensus.cc:348] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.124912 2638 raft_consensus.cc:374] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:32.124924 2638 raft_consensus.cc:729] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0287e6b3344e0997af29d050222b1, State: Initialized, Role: FOLLOWER
I20260925 13:17:32.124954 2638 consensus_queue.cc:260] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.125033 2638 ts_tablet_manager.cc:1428] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:32.125113 2642 tablet_replica.cc:439] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } }
I20260925 13:17:32.125207 2642 tablet_replica.cc:442] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 13:17:32.156080 2337 tablet.cc:2369] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:17:32.215705 2617 tablet.cc:2369] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:17:32.230589 2643 raft_consensus.cc:482] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:32.230702 2643 raft_consensus.cc:504] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.230887 2643 leader_election.cc:288] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cec0287e6b3344e0997af29d050222b1 (127.0.98.1:39685), fdbfc877556b4e598b6add2316605c8e (127.0.98.2:44693)
I20260925 13:17:32.231141 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fb69ce475e24c2b8bfeb1e0484dfd64" candidate_uuid: "d623e6a106a54352b3d1510ad55859b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0287e6b3344e0997af29d050222b1" is_pre_election: true
I20260925 13:17:32.231158 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fb69ce475e24c2b8bfeb1e0484dfd64" candidate_uuid: "d623e6a106a54352b3d1510ad55859b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdbfc877556b4e598b6add2316605c8e" is_pre_election: true
I20260925 13:17:32.231246 2430 raft_consensus.cc:2454] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d623e6a106a54352b3d1510ad55859b1 in term 0.
I20260925 13:17:32.231246 2290 raft_consensus.cc:2454] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d623e6a106a54352b3d1510ad55859b1 in term 0.
I20260925 13:17:32.231401 2502 leader_election.cc:302] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d623e6a106a54352b3d1510ad55859b1, fdbfc877556b4e598b6add2316605c8e; no voters:
I20260925 13:17:32.231484 2643 raft_consensus.cc:2781] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:32.231554 2643 raft_consensus.cc:482] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:32.231580 2643 raft_consensus.cc:3037] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:32.232270 2643 raft_consensus.cc:504] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.232375 2643 leader_election.cc:288] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [CANDIDATE]: Term 1 election: Requested vote from peers cec0287e6b3344e0997af29d050222b1 (127.0.98.1:39685), fdbfc877556b4e598b6add2316605c8e (127.0.98.2:44693)
I20260925 13:17:32.232519 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fb69ce475e24c2b8bfeb1e0484dfd64" candidate_uuid: "d623e6a106a54352b3d1510ad55859b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0287e6b3344e0997af29d050222b1"
I20260925 13:17:32.232586 2290 raft_consensus.cc:3037] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:32.232590 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fb69ce475e24c2b8bfeb1e0484dfd64" candidate_uuid: "d623e6a106a54352b3d1510ad55859b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdbfc877556b4e598b6add2316605c8e"
I20260925 13:17:32.232671 2430 raft_consensus.cc:3037] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:32.233158 2290 raft_consensus.cc:2454] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d623e6a106a54352b3d1510ad55859b1 in term 1.
I20260925 13:17:32.233326 2430 raft_consensus.cc:2454] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d623e6a106a54352b3d1510ad55859b1 in term 1.
I20260925 13:17:32.233328 2502 leader_election.cc:302] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0287e6b3344e0997af29d050222b1, d623e6a106a54352b3d1510ad55859b1; no voters:
I20260925 13:17:32.233426 2643 raft_consensus.cc:2781] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:32.233484 2643 raft_consensus.cc:686] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [term 1 LEADER]: Becoming Leader. State: Replica: d623e6a106a54352b3d1510ad55859b1, State: Running, Role: LEADER
I20260925 13:17:32.233568 2643 consensus_queue.cc:237] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.233850 2643 tablet_replica.cc:439] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1: TxnStatusTablet state changed. Reason: New leader d623e6a106a54352b3d1510ad55859b1. Latest consensus state: current_term: 1 leader_uuid: "d623e6a106a54352b3d1510ad55859b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } }
I20260925 13:17:32.233903 2643 tablet_replica.cc:442] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:17:32.234151 2658 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 13:17:32.234285 2658 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 13:17:32.234198 2132 catalog_manager.cc:5495] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d623e6a106a54352b3d1510ad55859b1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d623e6a106a54352b3d1510ad55859b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } health_report { overall_health: HEALTHY } } }
I20260925 13:17:32.240574 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.237948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45041 (local address 127.0.0.1:52626)
0925 13:17:32.238023 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:32.238033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:32.238062 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:32.238232 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:32.238233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:32.238234 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:17:32.238284 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.238286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.239145 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.239145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.239698 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.239699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.240152 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.240152 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.240195 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.240208 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:17:32.240456 (+ 248us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:17:32.240459 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:32.240482 (+ 23us) client_negotiation.cc:240] Negotiation successful
0925 13:17:32.240517 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 13:17:32.240592 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.237997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.0.98.3:45041)
0925 13:17:32.238041 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:32.238043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:32.238072 (+ 29us) server_negotiation.cc:407] Connection header received
0925 13:17:32.238128 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:32.238129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:32.238157 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:32.238227 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:32.238327 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.239069 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.239776 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.240076 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.240155 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.240281 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:32.240399 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:32.240480 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:32.240497 (+ 17us) server_negotiation.cc:299] Negotiation successful
0925 13:17:32.240520 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:17:32.242287 2290 raft_consensus.cc:1261] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1 [term 1 FOLLOWER]: Refusing update from remote peer d623e6a106a54352b3d1510ad55859b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:32.242300 2430 raft_consensus.cc:1261] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Refusing update from remote peer d623e6a106a54352b3d1510ad55859b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:32.242636 2646 consensus_queue.cc:1035] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:32.242763 2643 consensus_queue.cc:1035] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:32.244150 2642 tablet_replica.cc:439] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1: TxnStatusTablet state changed. Reason: New leader d623e6a106a54352b3d1510ad55859b1. Latest consensus state: current_term: 1 leader_uuid: "d623e6a106a54352b3d1510ad55859b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } }
I20260925 13:17:32.244253 2642 tablet_replica.cc:442] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:32.244546 2644 tablet_replica.cc:439] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e: TxnStatusTablet state changed. Reason: New leader d623e6a106a54352b3d1510ad55859b1. Latest consensus state: current_term: 1 leader_uuid: "d623e6a106a54352b3d1510ad55859b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } }
I20260925 13:17:32.244654 2644 tablet_replica.cc:442] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:32.245514 2646 tablet_replica.cc:439] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d623e6a106a54352b3d1510ad55859b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } }
I20260925 13:17:32.245639 2646 tablet_replica.cc:442] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:17:32.245743 2642 tablet_replica.cc:439] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d623e6a106a54352b3d1510ad55859b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } }
I20260925 13:17:32.245828 2642 tablet_replica.cc:442] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P cec0287e6b3344e0997af29d050222b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:32.245682 2646 tablet_replica.cc:439] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d623e6a106a54352b3d1510ad55859b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } }
I20260925 13:17:32.246022 2646 tablet_replica.cc:442] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P d623e6a106a54352b3d1510ad55859b1: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:17:32.246101 2662 mvcc.cc:205] Tried to move back new op lower bound from 7333241865182035968 to 7333241865149177856. Current Snapshot: MvccSnapshot[applied={T|T < 7333241865182035968}]
I20260925 13:17:32.246703 2644 tablet_replica.cc:439] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d623e6a106a54352b3d1510ad55859b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } } }
I20260925 13:17:32.246800 2644 tablet_replica.cc:442] T 7fb69ce475e24c2b8bfeb1e0484dfd64 P fdbfc877556b4e598b6add2316605c8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:32.247525 2663 mvcc.cc:205] Tried to move back new op lower bound from 7333241865182035968 to 7333241865149177856. Current Snapshot: MvccSnapshot[applied={T|T < 7333241865182035968}]
I20260925 13:17:32.250906 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.248157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.0.98.3:45041)
0925 13:17:32.248197 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:32.248199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:32.248209 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:17:32.248257 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:32.248257 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:32.248267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:32.248314 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:32.248411 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.249188 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.250338 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.250522 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.250577 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.250727 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:32.250753 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:32.250784 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:32.250835 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 13:17:32.250852 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:17:32.255990 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.253246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55019 (local address 127.0.98.3:45041)
0925 13:17:32.253287 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:32.253288 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:32.253614 (+ 326us) server_negotiation.cc:407] Connection header received
0925 13:17:32.253687 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:32.253687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:32.253695 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:32.253739 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:32.253853 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.254282 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.255365 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.255536 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.255587 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.255779 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:32.255802 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:32.255834 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:32.255898 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 13:17:32.255915 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:17:32.256067 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.253210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45041 (local address 127.0.98.3:55019)
0925 13:17:32.253558 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:32.253568 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:32.253620 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:32.253742 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:32.253743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:32.253744 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:17:32.253807 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.253809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.254343 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.254344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.255280 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.255283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.255599 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.255600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.255659 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.255678 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:17:32.255848 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:17:32.255849 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 13:17:32.255896 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 13:17:32.255944 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":3}
I20260925 13:17:32.258170 2430 raft_consensus.cc:1261] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Refusing update from remote peer d623e6a106a54352b3d1510ad55859b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:32.258163 2290 raft_consensus.cc:1261] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1 [term 1 FOLLOWER]: Refusing update from remote peer d623e6a106a54352b3d1510ad55859b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:32.258383 2666 consensus_queue.cc:1035] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:32.258490 2643 consensus_queue.cc:1035] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:32.259552 2661 mvcc.cc:205] Tried to move back new op lower bound from 7333241865248251904 to 7333241864495403008. Current Snapshot: MvccSnapshot[applied={T|T < 7333241865248251904}]
I20260925 13:17:32.270020 2132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:32.270141 2132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:32.270875 2132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7: Sending DeleteTablet for 3 replicas of tablet 143b7457b80848f68470b514f0742579
I20260925 13:17:32.271157 2410 tablet_service.cc:1514] Processing DeleteTablet for tablet 143b7457b80848f68470b514f0742579 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49068
I20260925 13:17:32.271180 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet 143b7457b80848f68470b514f0742579 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52622
I20260925 13:17:32.271219 2270 tablet_service.cc:1514] Processing DeleteTablet for tablet 143b7457b80848f68470b514f0742579 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43756
I20260925 13:17:32.271420 2678 tablet_replica.cc:330] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1: stopping tablet replica
I20260925 13:17:32.271410 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 13:17:32.271530 2678 raft_consensus.cc:2229] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 13:17:32.271551 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:32.271589 2678 raft_consensus.cc:2258] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:32.271747 2679 tablet_replica.cc:330] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1: stopping tablet replica
I20260925 13:17:32.271970 2679 raft_consensus.cc:2229] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:17:32.272082 2679 raft_consensus.cc:2258] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:32.272101 2678 ts_tablet_manager.cc:1905] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:32.272527 2679 ts_tablet_manager.cc:1905] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:32.272750 2550 tablet_service.cc:1467] Processing CreateTablet for tablet af19e3cf4d6d4016b97c47aef295479e (DEFAULT_TABLE table=test-table [id=88648c490eba4cd99494e65451a379c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:32.272835 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af19e3cf4d6d4016b97c47aef295479e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:32.273856 2678 ts_tablet_manager.cc:1918] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:17:32.273932 2678 log.cc:1196] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/wal/wals/143b7457b80848f68470b514f0742579
I20260925 13:17:32.273972 2637 tablet_bootstrap.cc:492] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1: Bootstrap starting.
I20260925 13:17:32.273996 2679 ts_tablet_manager.cc:1918] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:17:32.274044 2679 log.cc:1196] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/143b7457b80848f68470b514f0742579
I20260925 13:17:32.274173 2678 ts_tablet_manager.cc:1939] T 143b7457b80848f68470b514f0742579 P cec0287e6b3344e0997af29d050222b1: Deleting consensus metadata
I20260925 13:17:32.274278 2679 ts_tablet_manager.cc:1939] T 143b7457b80848f68470b514f0742579 P d623e6a106a54352b3d1510ad55859b1: Deleting consensus metadata
I20260925 13:17:32.274353 2637 tablet_bootstrap.cc:654] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:32.274933 2637 tablet_bootstrap.cc:492] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1: No bootstrap required, opened a new log
I20260925 13:17:32.274916 2410 tablet_service.cc:1467] Processing CreateTablet for tablet af19e3cf4d6d4016b97c47aef295479e (DEFAULT_TABLE table=test-table [id=88648c490eba4cd99494e65451a379c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:32.274935 2270 tablet_service.cc:1467] Processing CreateTablet for tablet af19e3cf4d6d4016b97c47aef295479e (DEFAULT_TABLE table=test-table [id=88648c490eba4cd99494e65451a379c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:32.274974 2637 ts_tablet_manager.cc:1397] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:32.275018 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af19e3cf4d6d4016b97c47aef295479e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:32.275020 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af19e3cf4d6d4016b97c47aef295479e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:32.275106 2637 raft_consensus.cc:348] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.275169 2637 raft_consensus.cc:374] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:32.275188 2637 raft_consensus.cc:729] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d623e6a106a54352b3d1510ad55859b1, State: Initialized, Role: FOLLOWER
I20260925 13:17:32.275238 2637 consensus_queue.cc:260] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.275315 2637 ts_tablet_manager.cc:1428] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:32.276178 2638 tablet_bootstrap.cc:492] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1: Bootstrap starting.
I20260925 13:17:32.276223 2636 tablet_bootstrap.cc:492] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e: Bootstrap starting.
I20260925 13:17:32.276503 2638 tablet_bootstrap.cc:654] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:32.276620 2636 tablet_bootstrap.cc:654] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:32.276877 2118 catalog_manager.cc:4841] TS cec0287e6b3344e0997af29d050222b1 (127.0.98.1:39685): tablet 143b7457b80848f68470b514f0742579 (table test-table [id=c8fe432b8ee146e29392d3ea1ca2e9e8]) successfully deleted
I20260925 13:17:32.276991 2118 catalog_manager.cc:4841] TS d623e6a106a54352b3d1510ad55859b1 (127.0.98.3:45041): tablet 143b7457b80848f68470b514f0742579 (table test-table [id=c8fe432b8ee146e29392d3ea1ca2e9e8]) successfully deleted
I20260925 13:17:32.277040 2638 tablet_bootstrap.cc:492] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1: No bootstrap required, opened a new log
I20260925 13:17:32.277073 2638 ts_tablet_manager.cc:1397] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:32.277202 2636 tablet_bootstrap.cc:492] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e: No bootstrap required, opened a new log
I20260925 13:17:32.277212 2638 raft_consensus.cc:348] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.277251 2636 ts_tablet_manager.cc:1397] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:32.277272 2638 raft_consensus.cc:374] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:32.277288 2638 raft_consensus.cc:729] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0287e6b3344e0997af29d050222b1, State: Initialized, Role: FOLLOWER
I20260925 13:17:32.277325 2638 consensus_queue.cc:260] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.277395 2638 ts_tablet_manager.cc:1428] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:32.277390 2636 raft_consensus.cc:348] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.277449 2636 raft_consensus.cc:374] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:32.277465 2636 raft_consensus.cc:729] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdbfc877556b4e598b6add2316605c8e, State: Initialized, Role: FOLLOWER
I20260925 13:17:32.277506 2636 consensus_queue.cc:260] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.277617 2636 ts_tablet_manager.cc:1428] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:32.281652 2677 tablet_replica.cc:330] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e: stopping tablet replica
I20260925 13:17:32.281807 2677 raft_consensus.cc:2229] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:32.281889 2677 raft_consensus.cc:2258] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:32.282351 2677 ts_tablet_manager.cc:1905] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:32.284210 2677 ts_tablet_manager.cc:1918] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:17:32.284286 2677 log.cc:1196] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/143b7457b80848f68470b514f0742579
I20260925 13:17:32.284500 2677 ts_tablet_manager.cc:1939] T 143b7457b80848f68470b514f0742579 P fdbfc877556b4e598b6add2316605c8e: Deleting consensus metadata
I20260925 13:17:32.284909 2118 catalog_manager.cc:4841] TS fdbfc877556b4e598b6add2316605c8e (127.0.98.2:44693): tablet 143b7457b80848f68470b514f0742579 (table test-table [id=c8fe432b8ee146e29392d3ea1ca2e9e8]) successfully deleted
I20260925 13:17:32.435969 2642 raft_consensus.cc:482] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:32.436071 2642 raft_consensus.cc:504] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.436383 2642 leader_election.cc:288] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdbfc877556b4e598b6add2316605c8e (127.0.98.2:44693), d623e6a106a54352b3d1510ad55859b1 (127.0.98.3:45041)
I20260925 13:17:32.439867 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.436757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45041 (local address 127.0.98.1:59567)
0925 13:17:32.436877 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:32.436884 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:32.436938 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:32.437141 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:32.437141 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:32.437142 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:32.437198 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.437199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.438121 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.438122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.439078 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.439081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.439592 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.439593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.439675 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.439680 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:17:32.439732 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 13:17:32.439773 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260925 13:17:32.439878 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.436852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59567 (local address 127.0.98.3:45041)
0925 13:17:32.436898 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:32.436900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:32.436931 (+ 31us) server_negotiation.cc:407] Connection header received
0925 13:17:32.436999 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:32.437000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:32.437009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:32.437087 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:32.437272 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.438035 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.439157 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.439522 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.439582 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.439594 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:32.439756 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 13:17:32.439808 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:17:32.440246 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af19e3cf4d6d4016b97c47aef295479e" candidate_uuid: "cec0287e6b3344e0997af29d050222b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d623e6a106a54352b3d1510ad55859b1" is_pre_election: true
I20260925 13:17:32.440387 2570 raft_consensus.cc:2454] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0287e6b3344e0997af29d050222b1 in term 0.
I20260925 13:17:32.440691 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.436676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33277 (local address 127.0.98.2:44693)
0925 13:17:32.436739 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:32.436741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:32.436868 (+ 127us) server_negotiation.cc:407] Connection header received
0925 13:17:32.436974 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:32.436975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:32.436985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:32.437066 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:32.437717 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.438369 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.439659 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.440151 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.440207 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.440217 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:32.440577 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 13:17:32.440620 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:17:32.440626 2222 leader_election.cc:302] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0287e6b3344e0997af29d050222b1, d623e6a106a54352b3d1510ad55859b1; no voters:
I20260925 13:17:32.440797 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.436587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44693 (local address 127.0.98.1:33277)
0925 13:17:32.436759 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:32.436809 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:32.436900 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:32.437084 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:32.437084 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:32.437087 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:32.437192 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.437195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.438465 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.438467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.439574 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.439578 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.440425 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.440426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.440487 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.440491 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:32.440567 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 13:17:32.440602 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":58,"threads_started":1}
I20260925 13:17:32.440807 2642 raft_consensus.cc:2781] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:32.440877 2642 raft_consensus.cc:482] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:32.440898 2642 raft_consensus.cc:3037] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:32.441311 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af19e3cf4d6d4016b97c47aef295479e" candidate_uuid: "cec0287e6b3344e0997af29d050222b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdbfc877556b4e598b6add2316605c8e" is_pre_election: true
I20260925 13:17:32.441414 2430 raft_consensus.cc:2454] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec0287e6b3344e0997af29d050222b1 in term 0.
I20260925 13:17:32.441638 2642 raft_consensus.cc:504] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.441754 2642 leader_election.cc:288] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [CANDIDATE]: Term 1 election: Requested vote from peers fdbfc877556b4e598b6add2316605c8e (127.0.98.2:44693), d623e6a106a54352b3d1510ad55859b1 (127.0.98.3:45041)
I20260925 13:17:32.441890 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af19e3cf4d6d4016b97c47aef295479e" candidate_uuid: "cec0287e6b3344e0997af29d050222b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdbfc877556b4e598b6add2316605c8e"
I20260925 13:17:32.441910 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af19e3cf4d6d4016b97c47aef295479e" candidate_uuid: "cec0287e6b3344e0997af29d050222b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d623e6a106a54352b3d1510ad55859b1"
I20260925 13:17:32.441970 2570 raft_consensus.cc:3037] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:32.441972 2430 raft_consensus.cc:3037] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:32.442543 2570 raft_consensus.cc:2454] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0287e6b3344e0997af29d050222b1 in term 1.
I20260925 13:17:32.442698 2222 leader_election.cc:302] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0287e6b3344e0997af29d050222b1, d623e6a106a54352b3d1510ad55859b1; no voters:
I20260925 13:17:32.442777 2642 raft_consensus.cc:2781] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:32.442808 2430 raft_consensus.cc:2454] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec0287e6b3344e0997af29d050222b1 in term 1.
I20260925 13:17:32.442906 2642 raft_consensus.cc:686] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [term 1 LEADER]: Becoming Leader. State: Replica: cec0287e6b3344e0997af29d050222b1, State: Running, Role: LEADER
I20260925 13:17:32.442977 2642 consensus_queue.cc:237] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:32.443693 2131 catalog_manager.cc:5495] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec0287e6b3344e0997af29d050222b1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cec0287e6b3344e0997af29d050222b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } health_report { overall_health: UNKNOWN } } }
I20260925 13:17:32.466046 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.463265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43768 (local address 127.0.98.1:39685)
0925 13:17:32.463315 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:32.463317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:32.463326 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:32.463400 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:32.463401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:32.463426 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:32.463489 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:32.463661 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.464152 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.465182 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.465367 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.465423 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.465656 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:32.465819 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:32.465897 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:32.465937 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 13:17:32.465963 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 13:17:32.471323 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.468804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42577 (local address 127.0.98.3:45041)
0925 13:17:32.468853 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:32.468855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:32.469011 (+ 156us) server_negotiation.cc:407] Connection header received
0925 13:17:32.469081 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:32.469082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:32.469090 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:32.469136 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:32.469277 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.469887 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.470692 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.470880 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.470933 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.471098 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:32.471132 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:32.471171 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:32.471231 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 13:17:32.471250 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:17:32.471350 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.468706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45041 (local address 127.0.98.1:42577)
0925 13:17:32.468952 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:32.468963 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:32.469025 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:32.469148 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:32.469149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:32.469150 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:17:32.469223 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.469225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.469972 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.469973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.470616 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.470619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.470958 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.470959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.471018 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.471031 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:17:32.471178 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:17:32.471179 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 13:17:32.471225 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 13:17:32.471267 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
I20260925 13:17:32.474509 2430 raft_consensus.cc:1261] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Refusing update from remote peer cec0287e6b3344e0997af29d050222b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:32.474571 2570 raft_consensus.cc:1261] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Refusing update from remote peer cec0287e6b3344e0997af29d050222b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:32.474781 2685 consensus_queue.cc:1035] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:32.474886 2642 consensus_queue.cc:1035] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:32.482939 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.480062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39685 (local address 127.0.98.3:48357)
0925 13:17:32.480115 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:32.480129 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:32.480176 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:32.480421 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:32.480422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:32.480423 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:17:32.480472 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.480473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.481238 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.481239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.482179 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:32.482182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.482533 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.482533 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:32.482585 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.482591 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:17:32.482763 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:17:32.482766 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:32.482809 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:17:32.482860 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 13:17:32.483093 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:32.480275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48357 (local address 127.0.98.1:39685)
0925 13:17:32.480319 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:32.480321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:32.480328 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:17:32.480345 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:32.480346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:32.480357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:32.480405 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:32.480543 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.481155 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.482249 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:32.482467 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:32.482518 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:32.482672 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:32.482708 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:32.482753 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:32.483010 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 13:17:32.483032 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 13:17:32.483537 2662 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 13:17:34.001197 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:34.001297 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:34.002110 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7: Sending DeleteTablet for 3 replicas of tablet af19e3cf4d6d4016b97c47aef295479e
I20260925 13:17:34.002338 2270 tablet_service.cc:1514] Processing DeleteTablet for tablet af19e3cf4d6d4016b97c47aef295479e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43756
I20260925 13:17:34.002403 2410 tablet_service.cc:1514] Processing DeleteTablet for tablet af19e3cf4d6d4016b97c47aef295479e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49068
I20260925 13:17:34.002514 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet af19e3cf4d6d4016b97c47aef295479e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52622
I20260925 13:17:34.002676 2717 tablet_replica.cc:330] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1: stopping tablet replica
I20260925 13:17:34.002720 2717 raft_consensus.cc:2229] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:34.002746 2717 raft_consensus.cc:2258] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:34.002878 2715 tablet_replica.cc:330] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1: stopping tablet replica
I20260925 13:17:34.002938 2715 raft_consensus.cc:2229] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:17:34.003010 2715 raft_consensus.cc:2258] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:34.003217 2715 ts_tablet_manager.cc:1905] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:34.004256 2717 ts_tablet_manager.cc:1905] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:34.004997 2716 tablet_replica.cc:330] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e: stopping tablet replica
I20260925 13:17:34.005064 2716 raft_consensus.cc:2229] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:34.005061 2715 ts_tablet_manager.cc:1918] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 13:17:34.005107 2716 raft_consensus.cc:2258] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:34.005111 2715 log.cc:1196] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/wal/wals/af19e3cf4d6d4016b97c47aef295479e
I20260925 13:17:34.005309 2715 ts_tablet_manager.cc:1939] T af19e3cf4d6d4016b97c47aef295479e P cec0287e6b3344e0997af29d050222b1: Deleting consensus metadata
I20260925 13:17:34.005316 2716 ts_tablet_manager.cc:1905] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:34.005720 2118 catalog_manager.cc:4841] TS cec0287e6b3344e0997af29d050222b1 (127.0.98.1:39685): tablet af19e3cf4d6d4016b97c47aef295479e (table test-table [id=88648c490eba4cd99494e65451a379c9]) successfully deleted
I20260925 13:17:34.005985 2717 ts_tablet_manager.cc:1918] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 13:17:34.006033 2717 log.cc:1196] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/af19e3cf4d6d4016b97c47aef295479e
I20260925 13:17:34.006206 2717 ts_tablet_manager.cc:1939] T af19e3cf4d6d4016b97c47aef295479e P d623e6a106a54352b3d1510ad55859b1: Deleting consensus metadata
I20260925 13:17:34.006780 2118 catalog_manager.cc:4841] TS d623e6a106a54352b3d1510ad55859b1 (127.0.98.3:45041): tablet af19e3cf4d6d4016b97c47aef295479e (table test-table [id=88648c490eba4cd99494e65451a379c9]) successfully deleted
I20260925 13:17:34.007042 2716 ts_tablet_manager.cc:1918] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 13:17:34.007102 2716 log.cc:1196] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/af19e3cf4d6d4016b97c47aef295479e
I20260925 13:17:34.007310 2716 ts_tablet_manager.cc:1939] T af19e3cf4d6d4016b97c47aef295479e P fdbfc877556b4e598b6add2316605c8e: Deleting consensus metadata
I20260925 13:17:34.007682 2118 catalog_manager.cc:4841] TS fdbfc877556b4e598b6add2316605c8e (127.0.98.2:44693): tablet af19e3cf4d6d4016b97c47aef295479e (table test-table [id=88648c490eba4cd99494e65451a379c9]) successfully deleted
Sep 25 13:17:34 dist-test-slave-t8d9 krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342254, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 13:17:34 dist-test-slave-t8d9 krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342254, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:34.031041 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:34.018777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.0.98.62:42007)
0925 13:17:34.018927 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:34.018928 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:34.018947 (+ 19us) server_negotiation.cc:407] Connection header received
0925 13:17:34.019025 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:34.019026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:34.019038 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:34.019100 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:34.019629 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.020144 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.020908 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.021110 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.021170 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:34.028520 (+ 7350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:34.028522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:34.028522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:34.028531 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:34.029906 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:34.030255 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:34.030257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:34.030257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:34.030319 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:34.030494 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:34.030495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:34.030495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:34.030619 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:34.030714 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:34.030838 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 13:17:34.030932 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260925 13:17:34.032418 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:17:34.032581 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:34.034026 2270 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd8201ff44e4b2da26d95c2bbcbd56f (DEFAULT_TABLE table=test-table [id=5bb69359decd41fe8f01d7d9825999aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:34.034026 2410 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd8201ff44e4b2da26d95c2bbcbd56f (DEFAULT_TABLE table=test-table [id=5bb69359decd41fe8f01d7d9825999aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:34.034046 2550 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd8201ff44e4b2da26d95c2bbcbd56f (DEFAULT_TABLE table=test-table [id=5bb69359decd41fe8f01d7d9825999aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:34.034166 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd8201ff44e4b2da26d95c2bbcbd56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:34.034166 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd8201ff44e4b2da26d95c2bbcbd56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:34.034183 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd8201ff44e4b2da26d95c2bbcbd56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:34.035351 2726 tablet_bootstrap.cc:492] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1: Bootstrap starting.
I20260925 13:17:34.035557 2727 tablet_bootstrap.cc:492] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e: Bootstrap starting.
I20260925 13:17:34.035842 2726 tablet_bootstrap.cc:654] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:34.035900 2727 tablet_bootstrap.cc:654] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:34.036228 2728 tablet_bootstrap.cc:492] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1: Bootstrap starting.
I20260925 13:17:34.036505 2726 tablet_bootstrap.cc:492] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1: No bootstrap required, opened a new log
I20260925 13:17:34.036553 2726 ts_tablet_manager.cc:1397] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:34.036597 2727 tablet_bootstrap.cc:492] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e: No bootstrap required, opened a new log
I20260925 13:17:34.036607 2728 tablet_bootstrap.cc:654] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:34.036650 2727 ts_tablet_manager.cc:1397] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:34.036689 2726 raft_consensus.cc:348] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.036774 2726 raft_consensus.cc:374] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:34.036795 2726 raft_consensus.cc:729] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d623e6a106a54352b3d1510ad55859b1, State: Initialized, Role: FOLLOWER
I20260925 13:17:34.036800 2727 raft_consensus.cc:348] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.036871 2727 raft_consensus.cc:374] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:34.036862 2726 consensus_queue.cc:260] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.036892 2727 raft_consensus.cc:729] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdbfc877556b4e598b6add2316605c8e, State: Initialized, Role: FOLLOWER
I20260925 13:17:34.036959 2726 ts_tablet_manager.cc:1428] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:34.036949 2727 consensus_queue.cc:260] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.037106 2727 ts_tablet_manager.cc:1428] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:34.037667 2728 tablet_bootstrap.cc:492] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1: No bootstrap required, opened a new log
I20260925 13:17:34.037721 2728 ts_tablet_manager.cc:1397] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:17:34.037832 2728 raft_consensus.cc:348] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.037906 2728 raft_consensus.cc:374] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:34.037923 2728 raft_consensus.cc:729] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0287e6b3344e0997af29d050222b1, State: Initialized, Role: FOLLOWER
I20260925 13:17:34.037969 2728 consensus_queue.cc:260] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.038064 2728 ts_tablet_manager.cc:1428] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:34.048218 2732 raft_consensus.cc:482] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:34.048309 2732 raft_consensus.cc:504] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.048635 2732 leader_election.cc:288] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cec0287e6b3344e0997af29d050222b1 (127.0.98.1:39685), d623e6a106a54352b3d1510ad55859b1 (127.0.98.3:45041)
I20260925 13:17:34.051816 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:34.048863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38125 (local address 127.0.98.1:39685)
0925 13:17:34.049017 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:34.049018 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:34.049028 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:17:34.049083 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:34.049084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:34.049095 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:34.049163 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:34.049313 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.049924 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.050969 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.051526 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.051603 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:34.051616 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:34.051727 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 13:17:34.051756 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260925 13:17:34.051867 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:34.048791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39685 (local address 127.0.98.2:38125)
0925 13:17:34.048955 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:34.048965 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:34.049016 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:34.049183 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:34.049184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:34.049186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:34.049245 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:34.049247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.050047 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.050049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:34.050885 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:34.050888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.051605 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.051606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:34.051669 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:34.051673 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:34.051719 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 13:17:34.051759 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
I20260925 13:17:34.052130 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd8201ff44e4b2da26d95c2bbcbd56f" candidate_uuid: "fdbfc877556b4e598b6add2316605c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0287e6b3344e0997af29d050222b1" is_pre_election: true
I20260925 13:17:34.052234 2290 raft_consensus.cc:2454] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdbfc877556b4e598b6add2316605c8e in term 0.
I20260925 13:17:34.052415 2362 leader_election.cc:302] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0287e6b3344e0997af29d050222b1, fdbfc877556b4e598b6add2316605c8e; no voters:
I20260925 13:17:34.052559 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:34.049007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45041 (local address 127.0.98.2:40117)
0925 13:17:34.049217 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:34.049227 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:34.049275 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:34.049412 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:34.049413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:34.049414 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:34.049456 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:34.049458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.050160 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.050161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:34.051525 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:34.051528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.052215 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.052240 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:34.052312 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:34.052315 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:34.052372 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 13:17:34.052407 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":66,"threads_started":1}
I20260925 13:17:34.052599 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:34.049066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40117 (local address 127.0.98.3:45041)
0925 13:17:34.049235 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:34.049237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:34.049316 (+ 79us) server_negotiation.cc:407] Connection header received
0925 13:17:34.049340 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:34.049341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:34.049351 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:34.049397 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:34.049512 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.050093 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.051605 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.052141 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.052204 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:34.052216 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:34.052490 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 13:17:34.052534 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260925 13:17:34.052703 2732 raft_consensus.cc:2781] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:34.052742 2732 raft_consensus.cc:482] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:34.052760 2732 raft_consensus.cc:3037] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:34.052758 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd8201ff44e4b2da26d95c2bbcbd56f" candidate_uuid: "fdbfc877556b4e598b6add2316605c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d623e6a106a54352b3d1510ad55859b1" is_pre_election: true
I20260925 13:17:34.052826 2570 raft_consensus.cc:2454] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdbfc877556b4e598b6add2316605c8e in term 0.
I20260925 13:17:34.053453 2732 raft_consensus.cc:504] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.053599 2732 leader_election.cc:288] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [CANDIDATE]: Term 1 election: Requested vote from peers cec0287e6b3344e0997af29d050222b1 (127.0.98.1:39685), d623e6a106a54352b3d1510ad55859b1 (127.0.98.3:45041)
I20260925 13:17:34.053763 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd8201ff44e4b2da26d95c2bbcbd56f" candidate_uuid: "fdbfc877556b4e598b6add2316605c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d623e6a106a54352b3d1510ad55859b1"
I20260925 13:17:34.053771 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd8201ff44e4b2da26d95c2bbcbd56f" candidate_uuid: "fdbfc877556b4e598b6add2316605c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0287e6b3344e0997af29d050222b1"
I20260925 13:17:34.053844 2570 raft_consensus.cc:3037] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:34.053844 2290 raft_consensus.cc:3037] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:34.054531 2570 raft_consensus.cc:2454] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdbfc877556b4e598b6add2316605c8e in term 1.
I20260925 13:17:34.054531 2290 raft_consensus.cc:2454] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdbfc877556b4e598b6add2316605c8e in term 1.
I20260925 13:17:34.054673 2362 leader_election.cc:302] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0287e6b3344e0997af29d050222b1, fdbfc877556b4e598b6add2316605c8e; no voters:
I20260925 13:17:34.054768 2732 raft_consensus.cc:2781] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:34.054903 2732 raft_consensus.cc:686] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [term 1 LEADER]: Becoming Leader. State: Replica: fdbfc877556b4e598b6add2316605c8e, State: Running, Role: LEADER
I20260925 13:17:34.054960 2732 consensus_queue.cc:237] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.055640 2131 catalog_manager.cc:5495] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e reported cstate change: term changed from 0 to 1, leader changed from <none> to fdbfc877556b4e598b6add2316605c8e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fdbfc877556b4e598b6add2316605c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } health_report { overall_health: UNKNOWN } } }
I20260925 13:17:34.066998 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:34.064784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49076 (local address 127.0.98.2:44693)
0925 13:17:34.064911 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:34.064913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:34.064921 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:17:34.064953 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:34.064953 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:34.064971 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:34.065037 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:34.065154 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.065670 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.066343 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.066531 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.066573 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:34.066730 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:34.066842 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:34.066893 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:34.066909 (+ 16us) server_negotiation.cc:299] Negotiation successful
0925 13:17:34.066923 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260925 13:17:34.070915 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:34.069023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60285 (local address 127.0.98.3:45041)
0925 13:17:34.069056 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:34.069058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:34.069102 (+ 44us) server_negotiation.cc:407] Connection header received
0925 13:17:34.069142 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:34.069143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:34.069152 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:34.069193 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:34.069293 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.069778 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.070446 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.070619 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.070663 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:34.070770 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:34.070791 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:34.070819 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:34.070849 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 13:17:34.070864 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 13:17:34.070943 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:34.068924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45041 (local address 127.0.98.2:60285)
0925 13:17:34.069048 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:34.069059 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:34.069107 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:34.069202 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:34.069203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:34.069204 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:17:34.069255 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:34.069256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.069832 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.069832 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:34.070390 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:34.070391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.070668 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.070668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:34.070709 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:34.070720 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:17:34.070819 (+ 99us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:17:34.070820 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 13:17:34.070846 (+ 26us) client_negotiation.cc:240] Negotiation successful
0925 13:17:34.070874 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":60,"threads_started":1}
I20260925 13:17:34.072696 2570 raft_consensus.cc:1261] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Refusing update from remote peer fdbfc877556b4e598b6add2316605c8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:34.072721 2290 raft_consensus.cc:1261] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1 [term 1 FOLLOWER]: Refusing update from remote peer fdbfc877556b4e598b6add2316605c8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:34.072929 2737 consensus_queue.cc:1035] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:34.073042 2732 consensus_queue.cc:1035] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:34.079857 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:34.077487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44693 (local address 127.0.98.3:47651)
0925 13:17:34.077617 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:34.077626 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:34.077716 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:34.077827 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:34.077828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:34.077829 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:17:34.077878 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:34.077879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.078489 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.078490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:34.079142 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:34.079145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.079499 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.079499 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:34.079553 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:34.079557 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:17:34.079695 (+ 138us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:17:34.079697 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:17:34.079735 (+ 38us) client_negotiation.cc:240] Negotiation successful
0925 13:17:34.079779 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":93,"threads_started":1}
I20260925 13:17:34.079851 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:34.077555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47651 (local address 127.0.98.2:44693)
0925 13:17:34.077588 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:34.077590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:34.077697 (+ 107us) server_negotiation.cc:407] Connection header received
0925 13:17:34.077764 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:34.077765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:34.077775 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:34.077825 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:34.077919 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.078381 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.079216 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.079445 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.079501 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:34.079613 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:34.079647 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:34.079689 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:34.079736 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 13:17:34.079774 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 13:17:34.086016 2133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:34.086112 2133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:34.086694 2133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7: Sending DeleteTablet for 3 replicas of tablet 1dd8201ff44e4b2da26d95c2bbcbd56f
I20260925 13:17:34.086920 2270 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd8201ff44e4b2da26d95c2bbcbd56f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43756
I20260925 13:17:34.086946 2410 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd8201ff44e4b2da26d95c2bbcbd56f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49068
I20260925 13:17:34.087013 2715 tablet_replica.cc:330] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1: stopping tablet replica
I20260925 13:17:34.087016 2716 tablet_replica.cc:330] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e: stopping tablet replica
I20260925 13:17:34.087054 2716 raft_consensus.cc:2229] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:17:34.087059 2715 raft_consensus.cc:2229] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:34.087091 2715 raft_consensus.cc:2258] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:34.087132 2716 raft_consensus.cc:2258] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:34.087134 2133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:17:34.087265 2133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:34.087422 2715 ts_tablet_manager.cc:1905] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:34.087602 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd8201ff44e4b2da26d95c2bbcbd56f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52622
I20260925 13:17:34.087693 2717 tablet_replica.cc:330] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1: stopping tablet replica
I20260925 13:17:34.087735 2717 raft_consensus.cc:2229] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:34.087769 2717 raft_consensus.cc:2258] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:34.088210 2716 ts_tablet_manager.cc:1905] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:34.088406 2270 tablet_service.cc:1467] Processing CreateTablet for tablet 9ebf785fa30f4f23a866195de7244ab8 (DEFAULT_TABLE table=test-table [id=f277ab6a39324f2e8822f118bc6e2810]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:34.088512 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebf785fa30f4f23a866195de7244ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:34.089088 2715 ts_tablet_manager.cc:1918] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:17:34.089133 2715 log.cc:1196] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-0/wal/wals/1dd8201ff44e4b2da26d95c2bbcbd56f
I20260925 13:17:34.089327 2715 ts_tablet_manager.cc:1939] T 1dd8201ff44e4b2da26d95c2bbcbd56f P cec0287e6b3344e0997af29d050222b1: Deleting consensus metadata
I20260925 13:17:34.089625 2118 catalog_manager.cc:4841] TS cec0287e6b3344e0997af29d050222b1 (127.0.98.1:39685): tablet 1dd8201ff44e4b2da26d95c2bbcbd56f (table test-table [id=5bb69359decd41fe8f01d7d9825999aa]) successfully deleted
I20260925 13:17:34.089722 2728 tablet_bootstrap.cc:492] T 9ebf785fa30f4f23a866195de7244ab8 P cec0287e6b3344e0997af29d050222b1: Bootstrap starting.
I20260925 13:17:34.089808 2410 tablet_service.cc:1467] Processing CreateTablet for tablet 9ebf785fa30f4f23a866195de7244ab8 (DEFAULT_TABLE table=test-table [id=f277ab6a39324f2e8822f118bc6e2810]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:34.089960 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebf785fa30f4f23a866195de7244ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:34.090023 2728 tablet_bootstrap.cc:654] T 9ebf785fa30f4f23a866195de7244ab8 P cec0287e6b3344e0997af29d050222b1: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:34.089963 2716 ts_tablet_manager.cc:1918] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:17:34.090135 2716 log.cc:1196] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/1dd8201ff44e4b2da26d95c2bbcbd56f
I20260925 13:17:34.090312 2716 ts_tablet_manager.cc:1939] T 1dd8201ff44e4b2da26d95c2bbcbd56f P fdbfc877556b4e598b6add2316605c8e: Deleting consensus metadata
I20260925 13:17:34.090473 2728 tablet_bootstrap.cc:492] T 9ebf785fa30f4f23a866195de7244ab8 P cec0287e6b3344e0997af29d050222b1: No bootstrap required, opened a new log
I20260925 13:17:34.090509 2728 ts_tablet_manager.cc:1397] T 9ebf785fa30f4f23a866195de7244ab8 P cec0287e6b3344e0997af29d050222b1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:34.090590 2118 catalog_manager.cc:4841] TS fdbfc877556b4e598b6add2316605c8e (127.0.98.2:44693): tablet 1dd8201ff44e4b2da26d95c2bbcbd56f (table test-table [id=5bb69359decd41fe8f01d7d9825999aa]) successfully deleted
I20260925 13:17:34.090606 2728 raft_consensus.cc:348] T 9ebf785fa30f4f23a866195de7244ab8 P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.090668 2728 raft_consensus.cc:374] T 9ebf785fa30f4f23a866195de7244ab8 P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:34.090690 2728 raft_consensus.cc:729] T 9ebf785fa30f4f23a866195de7244ab8 P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec0287e6b3344e0997af29d050222b1, State: Initialized, Role: FOLLOWER
I20260925 13:17:34.090723 2728 consensus_queue.cc:260] T 9ebf785fa30f4f23a866195de7244ab8 P cec0287e6b3344e0997af29d050222b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.090797 2728 ts_tablet_manager.cc:1428] T 9ebf785fa30f4f23a866195de7244ab8 P cec0287e6b3344e0997af29d050222b1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:34.091071 2727 tablet_bootstrap.cc:492] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e: Bootstrap starting.
I20260925 13:17:34.091189 2717 ts_tablet_manager.cc:1905] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:34.091430 2727 tablet_bootstrap.cc:654] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:34.092001 2727 tablet_bootstrap.cc:492] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e: No bootstrap required, opened a new log
I20260925 13:17:34.092038 2727 ts_tablet_manager.cc:1397] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:34.092193 2727 raft_consensus.cc:348] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.092250 2727 raft_consensus.cc:374] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:34.092268 2727 raft_consensus.cc:729] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdbfc877556b4e598b6add2316605c8e, State: Initialized, Role: FOLLOWER
I20260925 13:17:34.092321 2727 consensus_queue.cc:260] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.092429 2727 ts_tablet_manager.cc:1428] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:34.092453 2717 ts_tablet_manager.cc:1918] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:17:34.092494 2717 log.cc:1196] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/1dd8201ff44e4b2da26d95c2bbcbd56f
I20260925 13:17:34.092520 2550 tablet_service.cc:1467] Processing CreateTablet for tablet 9ebf785fa30f4f23a866195de7244ab8 (DEFAULT_TABLE table=test-table [id=f277ab6a39324f2e8822f118bc6e2810]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:34.092621 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebf785fa30f4f23a866195de7244ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:34.092674 2717 ts_tablet_manager.cc:1939] T 1dd8201ff44e4b2da26d95c2bbcbd56f P d623e6a106a54352b3d1510ad55859b1: Deleting consensus metadata
I20260925 13:17:34.093192 2118 catalog_manager.cc:4841] TS d623e6a106a54352b3d1510ad55859b1 (127.0.98.3:45041): tablet 1dd8201ff44e4b2da26d95c2bbcbd56f (table test-table [id=5bb69359decd41fe8f01d7d9825999aa]) successfully deleted
I20260925 13:17:34.093756 2726 tablet_bootstrap.cc:492] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1: Bootstrap starting.
I20260925 13:17:34.094019 2726 tablet_bootstrap.cc:654] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:34.094511 2726 tablet_bootstrap.cc:492] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1: No bootstrap required, opened a new log
I20260925 13:17:34.094563 2726 ts_tablet_manager.cc:1397] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 13:17:34.094669 2726 raft_consensus.cc:348] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.094736 2726 raft_consensus.cc:374] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:34.094755 2726 raft_consensus.cc:729] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d623e6a106a54352b3d1510ad55859b1, State: Initialized, Role: FOLLOWER
I20260925 13:17:34.094798 2726 consensus_queue.cc:260] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.094885 2726 ts_tablet_manager.cc:1428] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:34.107106 2676 raft_consensus.cc:482] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:34.107206 2676 raft_consensus.cc:504] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.107362 2676 leader_election.cc:288] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cec0287e6b3344e0997af29d050222b1 (127.0.98.1:39685), fdbfc877556b4e598b6add2316605c8e (127.0.98.2:44693)
I20260925 13:17:34.107560 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebf785fa30f4f23a866195de7244ab8" candidate_uuid: "d623e6a106a54352b3d1510ad55859b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdbfc877556b4e598b6add2316605c8e" is_pre_election: true
I20260925 13:17:34.107543 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebf785fa30f4f23a866195de7244ab8" candidate_uuid: "d623e6a106a54352b3d1510ad55859b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0287e6b3344e0997af29d050222b1" is_pre_election: true
I20260925 13:17:34.107649 2430 raft_consensus.cc:2454] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d623e6a106a54352b3d1510ad55859b1 in term 0.
I20260925 13:17:34.107651 2290 raft_consensus.cc:2454] T 9ebf785fa30f4f23a866195de7244ab8 P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d623e6a106a54352b3d1510ad55859b1 in term 0.
I20260925 13:17:34.107846 2502 leader_election.cc:302] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0287e6b3344e0997af29d050222b1, d623e6a106a54352b3d1510ad55859b1; no voters:
I20260925 13:17:34.107981 2676 raft_consensus.cc:2781] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:34.108018 2676 raft_consensus.cc:482] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:34.108044 2676 raft_consensus.cc:3037] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:34.108889 2676 raft_consensus.cc:504] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.109027 2676 leader_election.cc:288] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [CANDIDATE]: Term 1 election: Requested vote from peers cec0287e6b3344e0997af29d050222b1 (127.0.98.1:39685), fdbfc877556b4e598b6add2316605c8e (127.0.98.2:44693)
I20260925 13:17:34.109195 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebf785fa30f4f23a866195de7244ab8" candidate_uuid: "d623e6a106a54352b3d1510ad55859b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec0287e6b3344e0997af29d050222b1"
I20260925 13:17:34.109211 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebf785fa30f4f23a866195de7244ab8" candidate_uuid: "d623e6a106a54352b3d1510ad55859b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdbfc877556b4e598b6add2316605c8e"
I20260925 13:17:34.109275 2290 raft_consensus.cc:3037] T 9ebf785fa30f4f23a866195de7244ab8 P cec0287e6b3344e0997af29d050222b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:34.109278 2430 raft_consensus.cc:3037] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:34.109985 2290 raft_consensus.cc:2454] T 9ebf785fa30f4f23a866195de7244ab8 P cec0287e6b3344e0997af29d050222b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d623e6a106a54352b3d1510ad55859b1 in term 1.
I20260925 13:17:34.110008 2430 raft_consensus.cc:2454] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d623e6a106a54352b3d1510ad55859b1 in term 1.
I20260925 13:17:34.110136 2502 leader_election.cc:302] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cec0287e6b3344e0997af29d050222b1, d623e6a106a54352b3d1510ad55859b1; no voters:
I20260925 13:17:34.110249 2676 raft_consensus.cc:2781] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:34.110299 2676 raft_consensus.cc:686] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [term 1 LEADER]: Becoming Leader. State: Replica: d623e6a106a54352b3d1510ad55859b1, State: Running, Role: LEADER
I20260925 13:17:34.110345 2676 consensus_queue.cc:237] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } }
I20260925 13:17:34.110869 2131 catalog_manager.cc:5495] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d623e6a106a54352b3d1510ad55859b1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d623e6a106a54352b3d1510ad55859b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d623e6a106a54352b3d1510ad55859b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45041 } health_report { overall_health: HEALTHY } } }
I20260925 13:17:34.120903 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:34.118447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.0.98.3:45041)
0925 13:17:34.118484 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:34.118485 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:34.118494 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:34.118535 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:34.118536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:34.118545 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:34.118608 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:34.118692 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.119472 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.120204 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:34.120500 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:34.120563 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:34.120689 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:34.120722 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:34.120770 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:34.120805 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 13:17:34.120824 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 13:17:34.122879 2290 raft_consensus.cc:1261] T 9ebf785fa30f4f23a866195de7244ab8 P cec0287e6b3344e0997af29d050222b1 [term 1 FOLLOWER]: Refusing update from remote peer d623e6a106a54352b3d1510ad55859b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:34.122920 2430 raft_consensus.cc:1261] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Refusing update from remote peer d623e6a106a54352b3d1510ad55859b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:34.123077 2701 consensus_queue.cc:1035] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec0287e6b3344e0997af29d050222b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:34.123195 2701 consensus_queue.cc:1035] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdbfc877556b4e598b6add2316605c8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 13:17:34.127288 2694 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 13:17:35.643882 2132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:35.643981 2132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:35.644767 2132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fbe722714724aa09164062b7d2002b7: Sending DeleteTablet for 3 replicas of tablet 9ebf785fa30f4f23a866195de7244ab8
I20260925 13:17:35.644995 2270 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ebf785fa30f4f23a866195de7244ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43756
I20260925 13:17:35.645027 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ebf785fa30f4f23a866195de7244ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52622
I20260925 13:17:35.645059 2410 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ebf785fa30f4f23a866195de7244ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49068
I20260925 13:17:35.645210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 2205
I20260925 13:17:35.647558 2778 tablet_replica.cc:330] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e: stopping tablet replica
I20260925 13:17:35.647624 2778 raft_consensus.cc:2229] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:35.647665 2778 raft_consensus.cc:2258] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:35.647763 2776 tablet_replica.cc:330] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1: stopping tablet replica
I20260925 13:17:35.647814 2776 raft_consensus.cc:2229] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:17:35.647873 2776 raft_consensus.cc:2258] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:35.647943 2778 ts_tablet_manager.cc:1905] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:35.649518 2778 ts_tablet_manager.cc:1918] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 13:17:35.649605 2778 log.cc:1196] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/9ebf785fa30f4f23a866195de7244ab8
I20260925 13:17:35.649808 2778 ts_tablet_manager.cc:1939] T 9ebf785fa30f4f23a866195de7244ab8 P fdbfc877556b4e598b6add2316605c8e: Deleting consensus metadata
I20260925 13:17:35.649849 2776 ts_tablet_manager.cc:1905] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:35.650151 2118 catalog_manager.cc:4841] TS fdbfc877556b4e598b6add2316605c8e (127.0.98.2:44693): tablet 9ebf785fa30f4f23a866195de7244ab8 (table test-table [id=f277ab6a39324f2e8822f118bc6e2810]) successfully deleted
I20260925 13:17:35.651036 2776 ts_tablet_manager.cc:1918] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 13:17:35.651081 2776 log.cc:1196] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/9ebf785fa30f4f23a866195de7244ab8
I20260925 13:17:35.651269 2776 ts_tablet_manager.cc:1939] T 9ebf785fa30f4f23a866195de7244ab8 P d623e6a106a54352b3d1510ad55859b1: Deleting consensus metadata
I20260925 13:17:35.651703 2118 catalog_manager.cc:4841] TS d623e6a106a54352b3d1510ad55859b1 (127.0.98.3:45041): tablet 9ebf785fa30f4f23a866195de7244ab8 (table test-table [id=f277ab6a39324f2e8822f118bc6e2810]) successfully deleted
W20260925 13:17:35.651881 2118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39685 (error 108)
I20260925 13:17:35.652019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 2345
I20260925 13:17:35.653520 2780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:35.653271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39685 (local address 127.0.0.1:43770)
0925 13:17:35.653427 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:35.653485 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
W20260925 13:17:35.653654 2118 catalog_manager.cc:4584] TS cec0287e6b3344e0997af29d050222b1 (127.0.98.1:39685): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ebf785fa30f4f23a866195de7244ab8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39685: connect: Connection refused (error 111)
I20260925 13:17:35.657182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 2485
I20260925 13:17:35.662287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 2101
2026-09-25T13:17:35Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5377 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:17:35 dist-test-slave-t8d9 krb5kdc[2783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:17:35 dist-test-slave-t8d9 krb5kdc[2783](info): set up 2 sockets
Sep 25 13:17:35 dist-test-slave-t8d9 krb5kdc[2783](info): commencing operation
krb5kdc: starting...
W20260925 13:17:36.617125 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:17:36 dist-test-slave-t8d9 krb5kdc[2783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:17:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:17:36.730583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45859
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:36.809304 2796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:36.809514 2796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:36.809571 2796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:36.810722 2796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:36.810765 2796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:36.810781 2796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:36.810797 2796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:36.810823 2796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:36.812326 2796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:36.812562 2796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:36.812757 2796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:36.817224 2804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:36.817368 2796 server_base.cc:1034] running on GCE node
W20260925 13:17:36.817324 2801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:36.817238 2802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:36.817718 2796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:36.817983 2796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:36.819131 2796 hybrid_clock.cc:648] HybridClock initialized: now 1790342256819116 us; error 31 us; skew 500 ppm
Sep 25 13:17:36 dist-test-slave-t8d9 krb5kdc[2783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:36.821374 2796 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:17:36.821830 2796 webserver.cc:460] Webserver started at http://127.0.98.62:42673/ using document root <none> and password file <none>
I20260925 13:17:36.822021 2796 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:36.822064 2796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:36.822188 2796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:36.823197 2796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "d7ad97c005e3417bb64b99425afd6ae0"
format_stamp: "Formatted at 2026-09-25 13:17:36 on dist-test-slave-t8d9"
server_key: "5cb2f815b4a02af3fd0c74005523dc8e"
server_key_iv: "9efddb9302d47d15f14d936e0d173f99"
server_key_version: "encryptionkey@0"
I20260925 13:17:36.823539 2796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7ad97c005e3417bb64b99425afd6ae0"
format_stamp: "Formatted at 2026-09-25 13:17:36 on dist-test-slave-t8d9"
server_key: "5cb2f815b4a02af3fd0c74005523dc8e"
server_key_iv: "9efddb9302d47d15f14d936e0d173f99"
server_key_version: "encryptionkey@0"
I20260925 13:17:36.824764 2796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:36.825440 2811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:36.825609 2796 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:36.825683 2796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "d7ad97c005e3417bb64b99425afd6ae0"
format_stamp: "Formatted at 2026-09-25 13:17:36 on dist-test-slave-t8d9"
server_key: "5cb2f815b4a02af3fd0c74005523dc8e"
server_key_iv: "9efddb9302d47d15f14d936e0d173f99"
server_key_version: "encryptionkey@0"
I20260925 13:17:36.825747 2796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:36.838376 2796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:36.840247 2796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:36.840392 2796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:36.845011 2796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45859
I20260925 13:17:36.845046 2863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45859 every 8 connection(s)
I20260925 13:17:36.845568 2796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:36.846199 2864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:36.848346 2864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0: Bootstrap starting.
I20260925 13:17:36.848872 2864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:36.849097 2864 log.cc:824] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:36.849750 2864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0: No bootstrap required, opened a new log
I20260925 13:17:36.851246 2864 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ad97c005e3417bb64b99425afd6ae0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45859 } }
I20260925 13:17:36.851384 2864 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:36.851413 2864 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ad97c005e3417bb64b99425afd6ae0, State: Initialized, Role: FOLLOWER
I20260925 13:17:36.851517 2864 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ad97c005e3417bb64b99425afd6ae0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45859 } }
I20260925 13:17:36.851564 2864 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:36.851588 2864 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:36.851617 2864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:36.852433 2864 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ad97c005e3417bb64b99425afd6ae0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45859 } }
I20260925 13:17:36.852520 2864 leader_election.cc:302] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7ad97c005e3417bb64b99425afd6ae0; no voters:
I20260925 13:17:36.852646 2864 leader_election.cc:288] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:36.852686 2867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:36.852840 2867 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [term 1 LEADER]: Becoming Leader. State: Replica: d7ad97c005e3417bb64b99425afd6ae0, State: Running, Role: LEADER
I20260925 13:17:36.852864 2864 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:36.852934 2867 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ad97c005e3417bb64b99425afd6ae0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45859 } }
I20260925 13:17:36.853263 2869 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7ad97c005e3417bb64b99425afd6ae0. Latest consensus state: current_term: 1 leader_uuid: "d7ad97c005e3417bb64b99425afd6ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ad97c005e3417bb64b99425afd6ae0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45859 } } }
I20260925 13:17:36.853260 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7ad97c005e3417bb64b99425afd6ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ad97c005e3417bb64b99425afd6ae0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45859 } } }
I20260925 13:17:36.853462 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:36.853657 2869 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:36.853863 2876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:36.854358 2876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:36.855181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 2796
I20260925 13:17:36.855258 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:36.855433 392 external_mini_cluster.cc:1428] Setting key 7698d23f9e8a00d9d7265e2a7f09f6a4
I20260925 13:17:36.855804 2876 catalog_manager.cc:1347] Generated new cluster ID: 118263905e194da5b826b40e19d2ec2b
I20260925 13:17:36.855855 2876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:17:36 dist-test-slave-t8d9 krb5kdc[2783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:36.864565 2876 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:36.865178 2876 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:36.868325 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:36.856065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.0.98.62:45859)
0925 13:17:36.856228 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:36.856237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:36.857818 (+ 1581us) server_negotiation.cc:407] Connection header received
0925 13:17:36.857951 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:36.857959 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:36.858043 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:36.858197 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:36.858853 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:36.860330 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:36.861349 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:36.862058 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:36.862176 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:36.864915 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:36.864920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:36.864920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:36.864932 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:36.866825 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:36.867209 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:36.867212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:36.867212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:36.867308 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:36.867476 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:36.867476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:36.867477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:36.867723 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:36.867861 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:36.867949 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 13:17:36.868091 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260925 13:17:36.871939 2876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7ad97c005e3417bb64b99425afd6ae0: Generated new TSK 0
I20260925 13:17:36.872128 2876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:17:36.897193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45859
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:36.976616 2890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:36.976828 2890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:36.976859 2890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:36.978037 2890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:36.978080 2890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:36.978116 2890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:17:36.979633 2890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:36.979879 2890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:36.980089 2890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:36.984766 2898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:36.984910 2890 server_base.cc:1034] running on GCE node
W20260925 13:17:36.984835 2896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:36.984762 2895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:36.985240 2890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:36.985455 2890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:36.986583 2890 hybrid_clock.cc:648] HybridClock initialized: now 1790342256986577 us; error 39 us; skew 500 ppm
Sep 25 13:17:36 dist-test-slave-t8d9 krb5kdc[2783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:36.988873 2890 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:17:36.989254 2890 webserver.cc:460] Webserver started at http://127.0.98.1:33759/ using document root <none> and password file <none>
I20260925 13:17:36.989434 2890 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:36.989471 2890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:36.989598 2890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:36.990584 2890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "96b5a86de61d41cda7324f2282730647"
format_stamp: "Formatted at 2026-09-25 13:17:36 on dist-test-slave-t8d9"
server_key: "9aefe0a8fc37b5f65f20eb949c5235e2"
server_key_iv: "df03384707d5e993796afc4a3a547b07"
server_key_version: "encryptionkey@0"
I20260925 13:17:36.990861 2890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "96b5a86de61d41cda7324f2282730647"
format_stamp: "Formatted at 2026-09-25 13:17:36 on dist-test-slave-t8d9"
server_key: "9aefe0a8fc37b5f65f20eb949c5235e2"
server_key_iv: "df03384707d5e993796afc4a3a547b07"
server_key_version: "encryptionkey@0"
I20260925 13:17:36.991981 2890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:17:36.992575 2905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:36.992712 2890 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:36.992794 2890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "96b5a86de61d41cda7324f2282730647"
format_stamp: "Formatted at 2026-09-25 13:17:36 on dist-test-slave-t8d9"
server_key: "9aefe0a8fc37b5f65f20eb949c5235e2"
server_key_iv: "df03384707d5e993796afc4a3a547b07"
server_key_version: "encryptionkey@0"
I20260925 13:17:36.992847 2890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:37.003504 2890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:37.005512 2890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:37.005657 2890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:37.005837 2890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:37.006114 2890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:37.006146 2890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:37.006172 2890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:37.006189 2890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:37.011669 2890 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38261
I20260925 13:17:37.011727 3018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38261 every 8 connection(s)
I20260925 13:17:37.012023 2890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 13:17:37 dist-test-slave-t8d9 krb5kdc[2783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:37.020555 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:37.012668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40043 (local address 127.0.98.62:45859)
0925 13:17:37.012717 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:37.012719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:37.012907 (+ 188us) server_negotiation.cc:407] Connection header received
0925 13:17:37.013896 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:37.013898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:37.013915 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:37.013961 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:37.014558 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:37.015072 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:37.016057 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:37.016279 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:37.016333 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:37.018279 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:37.018280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:37.018281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:37.018288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:37.019368 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:37.019709 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:37.019710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:37.019710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:37.019766 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:37.019964 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:37.019965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:37.019965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:37.020074 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:37.020126 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:37.020372 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 13:17:37.020464 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:17:37.020694 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:37.012640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45859 (local address 127.0.98.1:40043)
0925 13:17:37.012797 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:37.012831 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:37.013788 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:37.014044 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:37.014048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:37.014291 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:37.014474 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:37.014477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:37.015152 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:37.015154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:37.015977 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:37.015980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:37.016346 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:37.016346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:37.016467 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:37.016888 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:37.016896 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:37.018194 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:37.019439 (+ 1245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:37.019446 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:37.019448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:37.019647 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:37.019818 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:37.019819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:37.019819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:37.019913 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:37.020124 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:37.020130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:37.020260 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 13:17:37.020346 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 13:17:37.020490 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260925 13:17:37.021152 3019 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45859
I20260925 13:17:37.021242 3019 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:37.021411 3019 heartbeater.cc:511] Master 127.0.98.62:45859 requested a full tablet report, sending...
I20260925 13:17:37.021903 2823 ts_manager.cc:194] Registered new tserver with Master: 96b5a86de61d41cda7324f2282730647 (127.0.98.1:38261)
I20260925 13:17:37.022087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 2890
I20260925 13:17:37.022178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:17:37.022339 392 external_mini_cluster.cc:1428] Setting key b0c5ca82d61d9fdc750ac1beb6781fc8
I20260925 13:17:37.022619 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40043
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:17:37.048225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45859
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:37.127075 3024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:37.127301 3024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:37.127326 3024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:37.128530 3024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:37.128573 3024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:37.128599 3024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:17:37.130209 3024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:37.130429 3024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:37.130659 3024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:37.135423 3032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:37.135401 3030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:37.135591 3024 server_base.cc:1034] running on GCE node
W20260925 13:17:37.135406 3029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:37.135850 3024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:37.136081 3024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:37.137212 3024 hybrid_clock.cc:648] HybridClock initialized: now 1790342257137184 us; error 43 us; skew 500 ppm
Sep 25 13:17:37 dist-test-slave-t8d9 krb5kdc[2783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:37.139596 3024 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:17:37.140077 3024 webserver.cc:460] Webserver started at http://127.0.98.2:36489/ using document root <none> and password file <none>
I20260925 13:17:37.140340 3024 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:37.140401 3024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:37.140529 3024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:37.141700 3024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "675747a329cf422d99eff406c28fc63c"
format_stamp: "Formatted at 2026-09-25 13:17:37 on dist-test-slave-t8d9"
server_key: "4958343f11a93f661e9656237750e278"
server_key_iv: "837d476bcda4c72f0eacf8500cf005f4"
server_key_version: "encryptionkey@0"
I20260925 13:17:37.142016 3024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "675747a329cf422d99eff406c28fc63c"
format_stamp: "Formatted at 2026-09-25 13:17:37 on dist-test-slave-t8d9"
server_key: "4958343f11a93f661e9656237750e278"
server_key_iv: "837d476bcda4c72f0eacf8500cf005f4"
server_key_version: "encryptionkey@0"
I20260925 13:17:37.143220 3024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:37.143858 3039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:37.144025 3024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:37.144102 3024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "675747a329cf422d99eff406c28fc63c"
format_stamp: "Formatted at 2026-09-25 13:17:37 on dist-test-slave-t8d9"
server_key: "4958343f11a93f661e9656237750e278"
server_key_iv: "837d476bcda4c72f0eacf8500cf005f4"
server_key_version: "encryptionkey@0"
I20260925 13:17:37.144162 3024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:37.159040 3024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:37.160794 3024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:37.160930 3024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:37.161144 3024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:37.161442 3024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:37.161474 3024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:37.161504 3024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:37.161520 3024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:37.166993 3024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39811
I20260925 13:17:37.167033 3152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39811 every 8 connection(s)
I20260925 13:17:37.167342 3024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:17:37.172897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 3024
I20260925 13:17:37.172992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:17:37.173161 392 external_mini_cluster.cc:1428] Setting key 63721e153b83154c34bc7c095d7ac852
Sep 25 13:17:37 dist-test-slave-t8d9 krb5kdc[2783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:37.176501 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:37.168062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59481 (local address 127.0.98.62:45859)
0925 13:17:37.168108 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:37.168110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:37.168309 (+ 199us) server_negotiation.cc:407] Connection header received
0925 13:17:37.169249 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:37.169251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:37.169262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:37.169322 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:37.170013 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:37.170518 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:37.171483 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:37.171675 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:37.171738 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:37.173993 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:37.173995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:37.173995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:37.174003 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:37.175249 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:37.175626 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:37.175627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:37.175627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:37.175685 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:37.175884 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:37.175884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:37.175885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:37.176011 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:37.176072 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:37.176371 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 13:17:37.176435 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:17:37.176740 3155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:37.168038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45859 (local address 127.0.98.2:59481)
0925 13:17:37.168190 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:37.168227 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:37.169128 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:37.169395 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:37.169400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:37.169708 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:37.169926 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:37.169931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:37.170585 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:37.170586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:37.171403 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:37.171405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:37.171742 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:37.171743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:37.171855 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:37.172338 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:37.172347 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:37.173902 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:37.175346 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:37.175353 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:37.175354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:37.175567 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:37.175741 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:37.175741 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:37.175741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:37.175834 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:37.176068 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:37.176073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:37.176235 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 13:17:37.176357 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 13:17:37.176516 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260925 13:17:37.177299 3153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45859
I20260925 13:17:37.177390 3153 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:37.177605 3153 heartbeater.cc:511] Master 127.0.98.62:45859 requested a full tablet report, sending...
I20260925 13:17:37.178038 2828 ts_manager.cc:194] Registered new tserver with Master: 675747a329cf422d99eff406c28fc63c (127.0.98.2:39811)
I20260925 13:17:37.178548 2828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59481
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:17:37.196827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45859
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:37.277154 3158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:37.277366 3158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:37.277390 3158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:37.278556 3158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:37.278596 3158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:37.278622 3158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:17:37.280048 3158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:37.280229 3158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:37.280404 3158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:37.285122 3164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:37.285079 3163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:37.285079 3166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:37.285168 3158 server_base.cc:1034] running on GCE node
I20260925 13:17:37.285509 3158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:37.285753 3158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:37.286877 3158 hybrid_clock.cc:648] HybridClock initialized: now 1790342257286869 us; error 26 us; skew 500 ppm
Sep 25 13:17:37 dist-test-slave-t8d9 krb5kdc[2783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:37.289227 3158 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:17:37.289674 3158 webserver.cc:460] Webserver started at http://127.0.98.3:39555/ using document root <none> and password file <none>
I20260925 13:17:37.289870 3158 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:37.289913 3158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:37.290030 3158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:37.291074 3158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "9b56422b297a4af2a0860f886a7401d5"
format_stamp: "Formatted at 2026-09-25 13:17:37 on dist-test-slave-t8d9"
server_key: "4b693e6d20f30eef52691b5cb60cb86f"
server_key_iv: "4993dd036dd76a4a997a1006b04e4b18"
server_key_version: "encryptionkey@0"
I20260925 13:17:37.291396 3158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9b56422b297a4af2a0860f886a7401d5"
format_stamp: "Formatted at 2026-09-25 13:17:37 on dist-test-slave-t8d9"
server_key: "4b693e6d20f30eef52691b5cb60cb86f"
server_key_iv: "4993dd036dd76a4a997a1006b04e4b18"
server_key_version: "encryptionkey@0"
I20260925 13:17:37.292644 3158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:17:37.293274 3173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:37.293463 3158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:17:37.293591 3158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "9b56422b297a4af2a0860f886a7401d5"
format_stamp: "Formatted at 2026-09-25 13:17:37 on dist-test-slave-t8d9"
server_key: "4b693e6d20f30eef52691b5cb60cb86f"
server_key_iv: "4993dd036dd76a4a997a1006b04e4b18"
server_key_version: "encryptionkey@0"
I20260925 13:17:37.293677 3158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:37.324479 3158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:37.326887 3158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:37.327044 3158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:37.327293 3158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:37.327610 3158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:37.327651 3158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:37.327685 3158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:37.327714 3158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:37.333870 3158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43077
I20260925 13:17:37.333921 3286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43077 every 8 connection(s)
I20260925 13:17:37.334239 3158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:17:37 dist-test-slave-t8d9 krb5kdc[2783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:37.341765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 3158
I20260925 13:17:37.341856 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:17:37.342015 392 external_mini_cluster.cc:1428] Setting key 614314470ad924c5784331769c269245
I20260925 13:17:37.343231 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:37.334962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56763 (local address 127.0.98.62:45859)
0925 13:17:37.335012 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:37.335013 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:37.335192 (+ 179us) server_negotiation.cc:407] Connection header received
0925 13:17:37.336097 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:37.336098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:37.336110 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:37.336183 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:37.336782 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:37.337307 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:37.338340 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:37.338525 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:37.338577 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:37.340507 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:37.340508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:37.340509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:37.340515 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:37.341862 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:37.342283 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:37.342284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:37.342284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:37.342352 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:37.342630 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:37.342631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:37.342631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:37.342782 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:37.342841 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:37.343101 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 13:17:37.343169 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 13:17:37.343416 3289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:37.334939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45859 (local address 127.0.98.3:56763)
0925 13:17:37.335083 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:37.335117 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:37.336007 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:37.336240 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:37.336243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:37.336478 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:37.336673 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:37.336677 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:37.337397 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:37.337398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:37.338263 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:37.338266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:37.338587 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:37.338588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:37.338691 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:37.339125 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:37.339140 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:37.340431 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:37.341950 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:37.341955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:37.341957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:37.342193 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:37.342421 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:37.342421 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:37.342422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:37.342569 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:37.342842 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:37.342848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:37.342994 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 13:17:37.343076 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 13:17:37.343197 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260925 13:17:37.343816 3287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45859
I20260925 13:17:37.343902 3287 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:37.344043 3287 heartbeater.cc:511] Master 127.0.98.62:45859 requested a full tablet report, sending...
I20260925 13:17:37.344353 2828 ts_manager.cc:194] Registered new tserver with Master: 9b56422b297a4af2a0860f886a7401d5 (127.0.98.3:43077)
I20260925 13:17:37.344702 2828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56763
I20260925 13:17:37.353709 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 13:17:37.366601 2887 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:37.363668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.0.98.62:45859)
0925 13:17:37.363753 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:37.363755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:37.365726 (+ 1971us) server_negotiation.cc:407] Connection header received
0925 13:17:37.365788 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:37.365789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:37.365803 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:37.365879 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:37.366522 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46618: BlockingRecv error: recv got EOF from 127.0.0.1:46618 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 13:17:37.367551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 2890
I20260925 13:17:37.371764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 3024
I20260925 13:17:37.375753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 3158
I20260925 13:17:37.379453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 2796
2026-09-25T13:17:37Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1715 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:17:37 dist-test-slave-t8d9 krb5kdc[3299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:17:37 dist-test-slave-t8d9 krb5kdc[3299](info): set up 2 sockets
Sep 25 13:17:37 dist-test-slave-t8d9 krb5kdc[3299](info): commencing operation
krb5kdc: starting...
W20260925 13:17:38.337291 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:17:38 dist-test-slave-t8d9 krb5kdc[3299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:17:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:17:38.451678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37699
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:38.530136 3312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:38.530336 3312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:38.530356 3312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:38.531474 3312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:38.531515 3312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:38.531527 3312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:38.531538 3312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:38.531549 3312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:38.532989 3312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:38.533201 3312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:38.533401 3312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:38.537981 3318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:38.538017 3320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:38.538009 3317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:38.538044 3312 server_base.cc:1034] running on GCE node
I20260925 13:17:38.538367 3312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:38.538630 3312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:38.539788 3312 hybrid_clock.cc:648] HybridClock initialized: now 1790342258539768 us; error 33 us; skew 500 ppm
Sep 25 13:17:38 dist-test-slave-t8d9 krb5kdc[3299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:38.542146 3312 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:17:38.542594 3312 webserver.cc:460] Webserver started at http://127.0.98.62:45447/ using document root <none> and password file <none>
I20260925 13:17:38.542794 3312 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:38.542840 3312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:38.542958 3312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:38.544008 3312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "be1c5f0739b84dd5acf8c1fdef8b3c1c"
format_stamp: "Formatted at 2026-09-25 13:17:38 on dist-test-slave-t8d9"
server_key: "dc3f414bac7df76694e1032ce02be86a"
server_key_iv: "7332b441f084a2019d3348518ba88a66"
server_key_version: "encryptionkey@0"
I20260925 13:17:38.544329 3312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "be1c5f0739b84dd5acf8c1fdef8b3c1c"
format_stamp: "Formatted at 2026-09-25 13:17:38 on dist-test-slave-t8d9"
server_key: "dc3f414bac7df76694e1032ce02be86a"
server_key_iv: "7332b441f084a2019d3348518ba88a66"
server_key_version: "encryptionkey@0"
I20260925 13:17:38.545588 3312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:17:38.546273 3327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:38.546417 3312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:38.546497 3312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "be1c5f0739b84dd5acf8c1fdef8b3c1c"
format_stamp: "Formatted at 2026-09-25 13:17:38 on dist-test-slave-t8d9"
server_key: "dc3f414bac7df76694e1032ce02be86a"
server_key_iv: "7332b441f084a2019d3348518ba88a66"
server_key_version: "encryptionkey@0"
I20260925 13:17:38.546562 3312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:38.561024 3312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:38.562834 3312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:38.562965 3312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:38.566732 3312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37699
I20260925 13:17:38.566766 3379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37699 every 8 connection(s)
I20260925 13:17:38.567082 3312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:38.567853 3380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:38.570072 3380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c: Bootstrap starting.
I20260925 13:17:38.570580 3380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:38.570806 3380 log.cc:824] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:38.571395 3380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c: No bootstrap required, opened a new log
I20260925 13:17:38.572861 3380 raft_consensus.cc:348] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1c5f0739b84dd5acf8c1fdef8b3c1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37699 } }
I20260925 13:17:38.572992 3380 raft_consensus.cc:374] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:38.573014 3380 raft_consensus.cc:729] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1c5f0739b84dd5acf8c1fdef8b3c1c, State: Initialized, Role: FOLLOWER
I20260925 13:17:38.573119 3380 consensus_queue.cc:260] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1c5f0739b84dd5acf8c1fdef8b3c1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37699 } }
I20260925 13:17:38.573197 3380 raft_consensus.cc:388] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:38.573225 3380 raft_consensus.cc:482] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:38.573257 3380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:38.573976 3380 raft_consensus.cc:504] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1c5f0739b84dd5acf8c1fdef8b3c1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37699 } }
I20260925 13:17:38.574057 3380 leader_election.cc:302] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be1c5f0739b84dd5acf8c1fdef8b3c1c; no voters:
I20260925 13:17:38.574169 3380 leader_election.cc:288] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:38.574226 3383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:38.574383 3383 raft_consensus.cc:686] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [term 1 LEADER]: Becoming Leader. State: Replica: be1c5f0739b84dd5acf8c1fdef8b3c1c, State: Running, Role: LEADER
I20260925 13:17:38.574404 3380 sys_catalog.cc:564] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:38.574481 3383 consensus_queue.cc:237] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1c5f0739b84dd5acf8c1fdef8b3c1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37699 } }
I20260925 13:17:38.574862 3385 sys_catalog.cc:455] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader be1c5f0739b84dd5acf8c1fdef8b3c1c. Latest consensus state: current_term: 1 leader_uuid: "be1c5f0739b84dd5acf8c1fdef8b3c1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1c5f0739b84dd5acf8c1fdef8b3c1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37699 } } }
I20260925 13:17:38.574952 3385 sys_catalog.cc:458] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:38.575135 3392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:38.574849 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be1c5f0739b84dd5acf8c1fdef8b3c1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1c5f0739b84dd5acf8c1fdef8b3c1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37699 } } }
I20260925 13:17:38.575505 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:38.575644 3392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:38.576133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 3312
I20260925 13:17:38.576224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:38.576380 392 external_mini_cluster.cc:1428] Setting key f6156b618657dd4cbecb2906ca01c240
I20260925 13:17:38.577378 3392 catalog_manager.cc:1347] Generated new cluster ID: 21d03011ea0340a89174d51809203f0c
I20260925 13:17:38.577425 3392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:17:38 dist-test-slave-t8d9 krb5kdc[3299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:38.585987 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:38.576916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.0.98.62:37699)
0925 13:17:38.577058 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:38.577067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:38.577098 (+ 31us) server_negotiation.cc:407] Connection header received
0925 13:17:38.577217 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:38.577224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:38.577281 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:38.577448 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:38.578059 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:38.578977 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:38.579687 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:38.580119 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:38.580213 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:38.582914 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:38.582919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:38.582919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:38.582932 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:38.584472 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:38.584807 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:38.584810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:38.584811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:38.584910 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:38.585086 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:38.585086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:38.585087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:38.585353 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:38.585502 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:38.585627 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 13:17:38.585765 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
I20260925 13:17:38.586813 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:17:38.587976 3392 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:38.588661 3392 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:38.600159 3392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be1c5f0739b84dd5acf8c1fdef8b3c1c: Generated new TSK 0
I20260925 13:17:38.600455 3392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:17:38.612932 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:38.605324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32800 (local address 127.0.98.62:37699)
0925 13:17:38.605364 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:38.605366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:38.605688 (+ 322us) server_negotiation.cc:407] Connection header received
0925 13:17:38.605719 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:38.605720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:38.605732 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:38.605777 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:38.609232 (+ 3455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:38.610079 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:38.611201 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:38.611528 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:38.611597 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:38.612374 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:38.612376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:38.612376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:38.612643 (+ 267us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:38.612666 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:38.612673 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:38.612739 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:38.612816 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 13:17:38.612847 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:17:38.615599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 3312
2026-09-25T13:17:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1238 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:17:38 dist-test-slave-t8d9 krb5kdc[3413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:17:38 dist-test-slave-t8d9 krb5kdc[3413](info): set up 2 sockets
Sep 25 13:17:38 dist-test-slave-t8d9 krb5kdc[3413](info): commencing operation
krb5kdc: starting...
W20260925 13:17:39.569180 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:17:39 dist-test-slave-t8d9 krb5kdc[3413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:17:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:17:39.682232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37399
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:39.762723 3426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:39.762949 3426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:39.762974 3426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:39.764190 3426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:39.764245 3426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:39.764259 3426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:39.764271 3426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:39.764281 3426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:39.765847 3426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:39.766062 3426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:39.766245 3426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:39.770845 3434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:39.770870 3431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:39.770939 3426 server_base.cc:1034] running on GCE node
W20260925 13:17:39.770891 3432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:39.771308 3426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:39.771584 3426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:39.772723 3426 hybrid_clock.cc:648] HybridClock initialized: now 1790342259772705 us; error 37 us; skew 500 ppm
Sep 25 13:17:39 dist-test-slave-t8d9 krb5kdc[3413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:39.782449 3426 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:17:39.782928 3426 webserver.cc:460] Webserver started at http://127.0.98.62:46295/ using document root <none> and password file <none>
I20260925 13:17:39.783159 3426 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:39.783209 3426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:39.783344 3426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:39.784490 3426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "ea5b70d65f584ba3b5a44f33dfea5c33"
format_stamp: "Formatted at 2026-09-25 13:17:39 on dist-test-slave-t8d9"
server_key: "4cf8a4c83e2bb8d87e7632e73e1bc59f"
server_key_iv: "77516f21dafef2eb606ccf033ac412a3"
server_key_version: "encryptionkey@0"
I20260925 13:17:39.784822 3426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea5b70d65f584ba3b5a44f33dfea5c33"
format_stamp: "Formatted at 2026-09-25 13:17:39 on dist-test-slave-t8d9"
server_key: "4cf8a4c83e2bb8d87e7632e73e1bc59f"
server_key_iv: "77516f21dafef2eb606ccf033ac412a3"
server_key_version: "encryptionkey@0"
I20260925 13:17:39.786336 3426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:39.787236 3441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:39.787460 3426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:39.787562 3426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "ea5b70d65f584ba3b5a44f33dfea5c33"
format_stamp: "Formatted at 2026-09-25 13:17:39 on dist-test-slave-t8d9"
server_key: "4cf8a4c83e2bb8d87e7632e73e1bc59f"
server_key_iv: "77516f21dafef2eb606ccf033ac412a3"
server_key_version: "encryptionkey@0"
I20260925 13:17:39.787629 3426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:39.820194 3426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:39.822197 3426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:39.822371 3426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:39.826007 3426 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37399
I20260925 13:17:39.826035 3493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37399 every 8 connection(s)
I20260925 13:17:39.826380 3426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:39.827046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 3426
I20260925 13:17:39.827137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:39.827301 392 external_mini_cluster.cc:1428] Setting key 66d28ee2140192f2545c18cd1431efb5
I20260925 13:17:39.827409 3494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:39.830780 3494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33: Bootstrap starting.
I20260925 13:17:39.831383 3494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:39.831697 3494 log.cc:824] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:39.832427 3494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33: No bootstrap required, opened a new log
Sep 25 13:17:39 dist-test-slave-t8d9 krb5kdc[3413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:39.834525 3494 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5b70d65f584ba3b5a44f33dfea5c33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37399 } }
I20260925 13:17:39.834673 3494 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:39.834758 3494 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5b70d65f584ba3b5a44f33dfea5c33, State: Initialized, Role: FOLLOWER
I20260925 13:17:39.834908 3494 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5b70d65f584ba3b5a44f33dfea5c33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37399 } }
I20260925 13:17:39.835007 3494 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:39.835050 3494 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:39.835100 3494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:39.835852 3494 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5b70d65f584ba3b5a44f33dfea5c33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37399 } }
I20260925 13:17:39.835963 3494 leader_election.cc:302] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea5b70d65f584ba3b5a44f33dfea5c33; no voters:
I20260925 13:17:39.836133 3494 leader_election.cc:288] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:39.836175 3499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:39.836337 3499 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [term 1 LEADER]: Becoming Leader. State: Replica: ea5b70d65f584ba3b5a44f33dfea5c33, State: Running, Role: LEADER
I20260925 13:17:39.836407 3494 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:39.836476 3499 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5b70d65f584ba3b5a44f33dfea5c33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37399 } }
I20260925 13:17:39.837216 3500 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea5b70d65f584ba3b5a44f33dfea5c33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5b70d65f584ba3b5a44f33dfea5c33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37399 } } }
I20260925 13:17:39.837306 3500 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:39.837849 3501 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea5b70d65f584ba3b5a44f33dfea5c33. Latest consensus state: current_term: 1 leader_uuid: "ea5b70d65f584ba3b5a44f33dfea5c33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5b70d65f584ba3b5a44f33dfea5c33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37399 } } }
I20260925 13:17:39.837921 3501 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33 [sys.catalog]: This master's current role is: LEADER
W20260925 13:17:39.838475 3515 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:17:39.838536 3515 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:17:39.838693 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:39.827789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.0.98.62:37399)
0925 13:17:39.827973 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:39.827983 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:39.828019 (+ 36us) server_negotiation.cc:407] Connection header received
0925 13:17:39.828587 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:39.828597 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:39.828665 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:39.828829 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:39.829427 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:39.830792 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:39.831591 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:39.832119 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:39.832248 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:39.834751 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:39.834756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:39.834756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:39.834768 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:39.836620 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:39.837155 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:39.837159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:39.837159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:39.837287 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:39.837512 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:39.837513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:39.837514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:39.837906 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:39.838070 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:39.838348 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 13:17:39.838495 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
I20260925 13:17:39.838900 3508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:39.839114 3508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:39.840737 3508 catalog_manager.cc:1347] Generated new cluster ID: 71c283943b944a08a3e8ad98e7ad8dc6
I20260925 13:17:39.840796 3508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:17:39.860177 3508 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:39.860935 3508 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:39.870011 3508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea5b70d65f584ba3b5a44f33dfea5c33: Generated new TSK 0
I20260925 13:17:39.870273 3508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:17:39.905050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37399
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:39.986521 3520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:39.986747 3520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:39.986779 3520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:39.987989 3520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:39.988037 3520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:39.988075 3520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:17:39.989590 3520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:39.989856 3520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:39.990074 3520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:39.995005 3528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:39.995021 3526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:39.995081 3525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:39.995270 3520 server_base.cc:1034] running on GCE node
I20260925 13:17:39.995512 3520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:39.995757 3520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:39.996902 3520 hybrid_clock.cc:648] HybridClock initialized: now 1790342259996891 us; error 33 us; skew 500 ppm
Sep 25 13:17:39 dist-test-slave-t8d9 krb5kdc[3413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:39.999393 3520 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:17:39.999878 3520 webserver.cc:460] Webserver started at http://127.0.98.1:38139/ using document root <none> and password file <none>
I20260925 13:17:40.000089 3520 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:40.000136 3520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:40.000285 3520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:40.001354 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "5c1d0464bff6470abf09da8fffe7afda"
format_stamp: "Formatted at 2026-09-25 13:17:39 on dist-test-slave-t8d9"
server_key: "9c8bb0089c71fcb0982b36b60a6f28af"
server_key_iv: "1e09ce13f19df69cf5a03444f1f03e01"
server_key_version: "encryptionkey@0"
I20260925 13:17:40.001724 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5c1d0464bff6470abf09da8fffe7afda"
format_stamp: "Formatted at 2026-09-25 13:17:39 on dist-test-slave-t8d9"
server_key: "9c8bb0089c71fcb0982b36b60a6f28af"
server_key_iv: "1e09ce13f19df69cf5a03444f1f03e01"
server_key_version: "encryptionkey@0"
I20260925 13:17:40.003041 3520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:17:40.003842 3535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:40.003993 3520 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:17:40.004082 3520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "5c1d0464bff6470abf09da8fffe7afda"
format_stamp: "Formatted at 2026-09-25 13:17:39 on dist-test-slave-t8d9"
server_key: "9c8bb0089c71fcb0982b36b60a6f28af"
server_key_iv: "1e09ce13f19df69cf5a03444f1f03e01"
server_key_version: "encryptionkey@0"
I20260925 13:17:40.004144 3520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:40.018909 3520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:40.021126 3520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:40.021289 3520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:40.021516 3520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:40.021862 3520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:40.021894 3520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:40.021924 3520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:40.021947 3520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:40.027909 3520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43945
I20260925 13:17:40.027999 3648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43945 every 8 connection(s)
I20260925 13:17:40.028311 3520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:17:40.030287 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 3520
I20260925 13:17:40.030390 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:17:40.030555 392 external_mini_cluster.cc:1428] Setting key b6a19a22b65bd69ab2011c9c20450285
Sep 25 13:17:40 dist-test-slave-t8d9 krb5kdc[3413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 13:17:40.038673 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:40.029122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47777 (local address 127.0.98.62:37399)
0925 13:17:40.029204 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:40.029207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:40.029399 (+ 192us) server_negotiation.cc:407] Connection header received
0925 13:17:40.029880 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:40.029881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:40.029892 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:40.029954 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:40.031046 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:40.031599 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:40.032683 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:40.032894 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:40.032953 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:40.035514 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:40.035516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:40.035517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:40.035524 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:40.036940 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:40.037566 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:40.037567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:40.037568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:40.037632 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:40.037960 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:40.037961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:40.037961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:40.038083 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:40.038162 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:40.038478 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 13:17:40.038592 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 13:17:40.038916 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:40.029048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37399 (local address 127.0.98.1:47777)
0925 13:17:40.029213 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:40.029255 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:40.029771 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:40.030015 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:40.030019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:40.030664 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:40.030966 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:40.030974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:40.031740 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:40.031742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:40.032585 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:40.032588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:40.032978 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:40.032979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:40.033127 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:40.033788 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:40.033798 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:40.035394 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:40.037046 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:40.037057 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:40.037061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:40.037452 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:40.037717 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:40.037718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:40.037718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:40.037880 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:40.038137 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:40.038146 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:40.038345 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 13:17:40.038465 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 13:17:40.038666 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260925 13:17:40.039426 3649 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37399
I20260925 13:17:40.039569 3649 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:40.039759 3649 heartbeater.cc:511] Master 127.0.98.62:37399 requested a full tablet report, sending...
I20260925 13:17:40.040288 3458 ts_manager.cc:194] Registered new tserver with Master: 5c1d0464bff6470abf09da8fffe7afda (127.0.98.1:43945)
I20260925 13:17:40.040948 3458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:17:40.057600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37399
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:40.140189 3654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:40.140412 3654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:40.140437 3654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:40.141669 3654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:40.141722 3654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:40.141749 3654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:17:40.143258 3654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:40.143522 3654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:40.143756 3654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:40.148782 3660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:40.148800 3662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:40.148905 3659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:40.148938 3654 server_base.cc:1034] running on GCE node
I20260925 13:17:40.149266 3654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:40.149559 3654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:40.150712 3654 hybrid_clock.cc:648] HybridClock initialized: now 1790342260150696 us; error 36 us; skew 500 ppm
Sep 25 13:17:40 dist-test-slave-t8d9 krb5kdc[3413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:40.153322 3654 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:17:40.153818 3654 webserver.cc:460] Webserver started at http://127.0.98.2:43141/ using document root <none> and password file <none>
I20260925 13:17:40.154039 3654 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:40.154085 3654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:40.154208 3654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:40.155260 3654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "cd033964841e403a8a138064902ec203"
format_stamp: "Formatted at 2026-09-25 13:17:40 on dist-test-slave-t8d9"
server_key: "764ab0a57f472d7f48c651c189c2894d"
server_key_iv: "e46a2f8ce6480eef071952314de7dc20"
server_key_version: "encryptionkey@0"
I20260925 13:17:40.155623 3654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cd033964841e403a8a138064902ec203"
format_stamp: "Formatted at 2026-09-25 13:17:40 on dist-test-slave-t8d9"
server_key: "764ab0a57f472d7f48c651c189c2894d"
server_key_iv: "e46a2f8ce6480eef071952314de7dc20"
server_key_version: "encryptionkey@0"
I20260925 13:17:40.156952 3654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:40.157786 3669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:40.157960 3654 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:40.158053 3654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "cd033964841e403a8a138064902ec203"
format_stamp: "Formatted at 2026-09-25 13:17:40 on dist-test-slave-t8d9"
server_key: "764ab0a57f472d7f48c651c189c2894d"
server_key_iv: "e46a2f8ce6480eef071952314de7dc20"
server_key_version: "encryptionkey@0"
I20260925 13:17:40.158115 3654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:40.172909 3654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:40.175346 3654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:40.175468 3654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:40.175661 3654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:40.175984 3654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:40.176023 3654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:40.176045 3654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:40.176059 3654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:40.181496 3654 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35671
I20260925 13:17:40.181624 3782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35671 every 8 connection(s)
I20260925 13:17:40.181936 3654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:17:40.182864 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 3654
I20260925 13:17:40.182957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:17:40.183143 392 external_mini_cluster.cc:1428] Setting key 5c609a8f556d075562ec7beba3e8a367
Sep 25 13:17:40 dist-test-slave-t8d9 krb5kdc[3413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 13:17:40.192502 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:40.182727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58587 (local address 127.0.98.62:37399)
0925 13:17:40.182790 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:40.182793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:40.182943 (+ 150us) server_negotiation.cc:407] Connection header received
0925 13:17:40.183531 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:40.183532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:40.183543 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:40.183583 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:40.184544 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:40.185276 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:40.186716 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:40.186928 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:40.186993 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:40.189629 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:40.189631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:40.189631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:40.189639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:40.191096 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:40.191507 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:40.191513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:40.191514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:40.191577 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:40.191807 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:40.191808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:40.191808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:40.191931 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:40.191988 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:40.192313 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 13:17:40.192408 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:17:40.192690 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:40.182667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37399 (local address 127.0.98.2:58587)
0925 13:17:40.182823 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:40.182866 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:40.183446 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:40.183661 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:40.183665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:40.184255 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:40.184456 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:40.184460 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:40.185393 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:40.185395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:40.186603 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:40.186606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:40.187017 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:40.187019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:40.187179 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:40.187713 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:40.187723 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:40.189490 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:40.191183 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:40.191192 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:40.191195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:40.191435 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:40.191634 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:40.191635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:40.191635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:40.191749 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:40.191987 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:40.191994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:40.192178 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 13:17:40.192271 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 13:17:40.192446 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260925 13:17:40.193275 3783 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37399
I20260925 13:17:40.193382 3783 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:40.193610 3783 heartbeater.cc:511] Master 127.0.98.62:37399 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 13:17:40.194000 3458 ts_manager.cc:194] Registered new tserver with Master: cd033964841e403a8a138064902ec203 (127.0.98.2:35671)
I20260925 13:17:40.194378 3458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:17:40.208611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37399
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:40.291498 3788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:40.291726 3788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:40.291750 3788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:40.292954 3788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:40.292999 3788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:40.293025 3788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:17:40.294590 3788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:40.294848 3788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:40.295073 3788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:40.299808 3796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:40.299814 3793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:40.299998 3794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:40.300082 3788 server_base.cc:1034] running on GCE node
I20260925 13:17:40.300285 3788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:40.300537 3788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:40.301695 3788 hybrid_clock.cc:648] HybridClock initialized: now 1790342260301676 us; error 36 us; skew 500 ppm
Sep 25 13:17:40 dist-test-slave-t8d9 krb5kdc[3413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:40.304073 3788 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:17:40.304520 3788 webserver.cc:460] Webserver started at http://127.0.98.3:36419/ using document root <none> and password file <none>
I20260925 13:17:40.304736 3788 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:40.304783 3788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:40.304908 3788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:40.306078 3788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "10b8666f476845838fe46269b6ffa0c6"
format_stamp: "Formatted at 2026-09-25 13:17:40 on dist-test-slave-t8d9"
server_key: "93ebca756a73728eb54f64120dc79c52"
server_key_iv: "fa6a44225539d147702ea3ad690b2bbc"
server_key_version: "encryptionkey@0"
I20260925 13:17:40.306432 3788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "10b8666f476845838fe46269b6ffa0c6"
format_stamp: "Formatted at 2026-09-25 13:17:40 on dist-test-slave-t8d9"
server_key: "93ebca756a73728eb54f64120dc79c52"
server_key_iv: "fa6a44225539d147702ea3ad690b2bbc"
server_key_version: "encryptionkey@0"
I20260925 13:17:40.307760 3788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:17:40.308583 3803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:40.308833 3788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:40.308928 3788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "10b8666f476845838fe46269b6ffa0c6"
format_stamp: "Formatted at 2026-09-25 13:17:40 on dist-test-slave-t8d9"
server_key: "93ebca756a73728eb54f64120dc79c52"
server_key_iv: "fa6a44225539d147702ea3ad690b2bbc"
server_key_version: "encryptionkey@0"
I20260925 13:17:40.309001 3788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:40.326506 3788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:40.328981 3788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:40.329164 3788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:40.329440 3788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:40.329825 3788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:40.329862 3788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:40.329893 3788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:40.329909 3788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:40.335515 3788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36045
I20260925 13:17:40.335561 3916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36045 every 8 connection(s)
I20260925 13:17:40.335896 3788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:17:40 dist-test-slave-t8d9 krb5kdc[3413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:40.344008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 3788
I20260925 13:17:40.344120 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:17:40.344307 392 external_mini_cluster.cc:1428] Setting key b9c1e05f405958a49f654e3827edb678
I20260925 13:17:40.346446 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:40.336612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43917 (local address 127.0.98.62:37399)
0925 13:17:40.336668 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:40.336670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:40.336930 (+ 260us) server_negotiation.cc:407] Connection header received
0925 13:17:40.337463 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:40.337464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:40.337477 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:40.337518 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:40.338636 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:40.339142 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:40.340198 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:40.340417 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:40.340480 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:40.343132 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:40.343135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:40.343135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:40.343143 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:40.344735 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:40.345278 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:40.345279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:40.345280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:40.345349 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:40.345740 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:40.345741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:40.345741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:40.345869 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:40.345931 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:40.346295 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 13:17:40.346374 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 13:17:40.346602 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:40.336592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37399 (local address 127.0.98.3:43917)
0925 13:17:40.336800 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:40.336841 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:40.337356 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:40.337641 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:40.337645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:40.338304 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:40.338521 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:40.338527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:40.339245 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:40.339245 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:40.340093 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:40.340096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:40.340501 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:40.340503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:40.340631 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:40.341168 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:40.341178 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:40.343000 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:40.344817 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:40.344831 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:40.344834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:40.345198 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:40.345447 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:40.345448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:40.345448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:40.345642 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:40.345958 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:40.345966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:40.346119 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 13:17:40.346217 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 13:17:40.346377 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
I20260925 13:17:40.347255 3917 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37399
I20260925 13:17:40.347366 3917 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:40.347570 3917 heartbeater.cc:511] Master 127.0.98.62:37399 requested a full tablet report, sending...
I20260925 13:17:40.347966 3458 ts_manager.cc:194] Registered new tserver with Master: 10b8666f476845838fe46269b6ffa0c6 (127.0.98.3:36045)
I20260925 13:17:40.348364 3458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43917
I20260925 13:17:40.356386 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 13:17:40 dist-test-slave-t8d9 krb5kdc[3413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342260, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 13:17:40 dist-test-slave-t8d9 krb5kdc[3413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342260, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:40.379653 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:40.371693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.0.98.62:37399)
0925 13:17:40.371760 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:40.371763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:40.371881 (+ 118us) server_negotiation.cc:407] Connection header received
0925 13:17:40.371947 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:40.371948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:40.371959 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:40.372005 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:40.372578 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:40.373089 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:40.373896 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:40.374096 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:40.374156 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:40.376675 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:40.376688 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:40.376688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:40.376704 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:40.378189 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:40.378576 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:40.378577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:40.378577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:40.378641 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:40.379000 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:40.379001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:40.379002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:40.379187 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:40.379278 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:40.379492 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 13:17:40.379576 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 13:17:40.380470 3458 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54762
I20260925 13:17:40.381160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 3520
I20260925 13:17:40.385457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 3654
I20260925 13:17:40.389499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 3788
I20260925 13:17:40.393363 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 3426
2026-09-25T13:17:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1776 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:17:40 dist-test-slave-t8d9 krb5kdc[3929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:17:40 dist-test-slave-t8d9 krb5kdc[3929](info): set up 2 sockets
Sep 25 13:17:40 dist-test-slave-t8d9 krb5kdc[3929](info): commencing operation
krb5kdc: starting...
W20260925 13:17:41.345296 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:17:41 dist-test-slave-t8d9 krb5kdc[3929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:17:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:17:41.462317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35687
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:41.541960 3942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:41.542181 3942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:41.542201 3942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:41.543365 3942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:41.543401 3942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:41.543413 3942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:41.543424 3942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:41.543434 3942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:41.544869 3942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:41.545065 3942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:41.545245 3942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:41.549865 3948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:41.549850 3950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:41.549855 3947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:41.550112 3942 server_base.cc:1034] running on GCE node
I20260925 13:17:41.550346 3942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:41.550621 3942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:41.551779 3942 hybrid_clock.cc:648] HybridClock initialized: now 1790342261551752 us; error 37 us; skew 500 ppm
Sep 25 13:17:41 dist-test-slave-t8d9 krb5kdc[3929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:41.554301 3942 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:17:41.554745 3942 webserver.cc:460] Webserver started at http://127.0.98.62:37825/ using document root <none> and password file <none>
I20260925 13:17:41.554978 3942 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:41.555027 3942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:41.555156 3942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:41.556205 3942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "f4cc7c155e3a4d86a1d96d76a1f4d79b"
format_stamp: "Formatted at 2026-09-25 13:17:41 on dist-test-slave-t8d9"
server_key: "0fda38dfe095cfe8c8de4b6dd183354d"
server_key_iv: "9aeb44d712997f2a7fb084206ce244a2"
server_key_version: "encryptionkey@0"
I20260925 13:17:41.556550 3942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4cc7c155e3a4d86a1d96d76a1f4d79b"
format_stamp: "Formatted at 2026-09-25 13:17:41 on dist-test-slave-t8d9"
server_key: "0fda38dfe095cfe8c8de4b6dd183354d"
server_key_iv: "9aeb44d712997f2a7fb084206ce244a2"
server_key_version: "encryptionkey@0"
I20260925 13:17:41.557909 3942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:41.558665 3957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:41.558842 3942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:41.558928 3942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "f4cc7c155e3a4d86a1d96d76a1f4d79b"
format_stamp: "Formatted at 2026-09-25 13:17:41 on dist-test-slave-t8d9"
server_key: "0fda38dfe095cfe8c8de4b6dd183354d"
server_key_iv: "9aeb44d712997f2a7fb084206ce244a2"
server_key_version: "encryptionkey@0"
I20260925 13:17:41.558990 3942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:41.566653 3942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:41.568900 3942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:41.569032 3942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:41.572669 3942 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35687
I20260925 13:17:41.572749 4009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35687 every 8 connection(s)
I20260925 13:17:41.573042 3942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:41.573952 4010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:41.576140 4010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b: Bootstrap starting.
I20260925 13:17:41.576655 4010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:41.576910 4010 log.cc:824] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:41.576949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 3942
I20260925 13:17:41.577028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:41.577189 392 external_mini_cluster.cc:1428] Setting key 25f012f5cabfe5c2e2f46147fba91f67
I20260925 13:17:41.577630 4010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b: No bootstrap required, opened a new log
I20260925 13:17:41.579489 4010 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cc7c155e3a4d86a1d96d76a1f4d79b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35687 } }
I20260925 13:17:41.579632 4010 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:41.579664 4010 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4cc7c155e3a4d86a1d96d76a1f4d79b, State: Initialized, Role: FOLLOWER
I20260925 13:17:41.579793 4010 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cc7c155e3a4d86a1d96d76a1f4d79b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35687 } }
I20260925 13:17:41.579866 4010 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:41.579912 4010 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:41.579958 4010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:41.580494 4010 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cc7c155e3a4d86a1d96d76a1f4d79b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35687 } }
I20260925 13:17:41.580587 4010 leader_election.cc:302] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4cc7c155e3a4d86a1d96d76a1f4d79b; no voters:
I20260925 13:17:41.580732 4010 leader_election.cc:288] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:41.580777 4015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:41.580972 4010 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:41.580960 4015 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [term 1 LEADER]: Becoming Leader. State: Replica: f4cc7c155e3a4d86a1d96d76a1f4d79b, State: Running, Role: LEADER
I20260925 13:17:41.581215 4015 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cc7c155e3a4d86a1d96d76a1f4d79b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35687 } }
I20260925 13:17:41.581664 4017 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4cc7c155e3a4d86a1d96d76a1f4d79b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cc7c155e3a4d86a1d96d76a1f4d79b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35687 } } }
I20260925 13:17:41.581681 4016 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4cc7c155e3a4d86a1d96d76a1f4d79b. Latest consensus state: current_term: 1 leader_uuid: "f4cc7c155e3a4d86a1d96d76a1f4d79b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cc7c155e3a4d86a1d96d76a1f4d79b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35687 } } }
I20260925 13:17:41.581799 4016 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:41.581786 4017 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:41.582203 4024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:41.582727 4024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 13:17:41 dist-test-slave-t8d9 krb5kdc[3929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:41.585377 4024 catalog_manager.cc:1347] Generated new cluster ID: c14c945f49b1435283a31854e60686e9
I20260925 13:17:41.585426 4024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:17:41.588742 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:41.577682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.0.98.62:35687)
0925 13:17:41.577880 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:41.577888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:41.577913 (+ 25us) server_negotiation.cc:407] Connection header received
0925 13:17:41.578838 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:41.578847 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:41.578899 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:41.579029 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:41.579645 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:41.580988 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:41.581923 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:41.582530 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:41.582640 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:41.585299 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:41.585304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:41.585304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:41.585317 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:41.587176 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:41.587581 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:41.587584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:41.587584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:41.587683 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:41.587877 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:41.587878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:41.587878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:41.588155 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:41.588307 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:41.588416 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 13:17:41.588544 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260925 13:17:41.593282 4024 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:41.593859 4024 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:41.597338 4024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4cc7c155e3a4d86a1d96d76a1f4d79b: Generated new TSK 0
I20260925 13:17:41.597565 4024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:17:41.623795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35687
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:41.705235 4036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:41.705442 4036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:41.705473 4036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:41.706682 4036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:41.706736 4036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:41.706772 4036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:17:41.708571 4036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:41.708820 4036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:41.709035 4036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:41.713802 4041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:41.713891 4036 server_base.cc:1034] running on GCE node
W20260925 13:17:41.713810 4042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:41.713814 4044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:41.714241 4036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:41.714471 4036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:41.715642 4036 hybrid_clock.cc:648] HybridClock initialized: now 1790342261715629 us; error 33 us; skew 500 ppm
Sep 25 13:17:41 dist-test-slave-t8d9 krb5kdc[3929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:41.718461 4036 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:17:41.718940 4036 webserver.cc:460] Webserver started at http://127.0.98.1:43719/ using document root <none> and password file <none>
I20260925 13:17:41.719158 4036 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:41.719206 4036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:41.719328 4036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:41.720361 4036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "5b3c8b2f04b640d0a9496b0e675cbf5a"
format_stamp: "Formatted at 2026-09-25 13:17:41 on dist-test-slave-t8d9"
server_key: "7710c3022162cf2f9d15ee983f981df7"
server_key_iv: "9bb100c6cdfdc14cd9306ba01b7ba174"
server_key_version: "encryptionkey@0"
I20260925 13:17:41.720686 4036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5b3c8b2f04b640d0a9496b0e675cbf5a"
format_stamp: "Formatted at 2026-09-25 13:17:41 on dist-test-slave-t8d9"
server_key: "7710c3022162cf2f9d15ee983f981df7"
server_key_iv: "9bb100c6cdfdc14cd9306ba01b7ba174"
server_key_version: "encryptionkey@0"
I20260925 13:17:41.721982 4036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 13:17:41.722661 4051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:41.722858 4036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:41.722937 4036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "5b3c8b2f04b640d0a9496b0e675cbf5a"
format_stamp: "Formatted at 2026-09-25 13:17:41 on dist-test-slave-t8d9"
server_key: "7710c3022162cf2f9d15ee983f981df7"
server_key_iv: "9bb100c6cdfdc14cd9306ba01b7ba174"
server_key_version: "encryptionkey@0"
I20260925 13:17:41.722997 4036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:41.743225 4036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:41.745426 4036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:41.745604 4036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:41.745836 4036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:41.746129 4036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:41.746162 4036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:41.746192 4036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:41.746214 4036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:41.751431 4036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46773
I20260925 13:17:41.751492 4164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46773 every 8 connection(s)
I20260925 13:17:41.751801 4036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 13:17:41 dist-test-slave-t8d9 krb5kdc[3929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:41.759122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 4036
I20260925 13:17:41.759204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:17:41.759362 392 external_mini_cluster.cc:1428] Setting key 5d3ae9280b48e505b73fc4b215b237dd
I20260925 13:17:41.760859 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:41.752492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54277 (local address 127.0.98.62:35687)
0925 13:17:41.752554 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:41.752557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:41.752747 (+ 190us) server_negotiation.cc:407] Connection header received
0925 13:17:41.753677 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:41.753678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:41.753688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:41.753740 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:41.754365 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:41.754898 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:41.755926 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:41.756134 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:41.756190 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:41.758279 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:41.758281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:41.758281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:41.758288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:41.759539 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:41.759997 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:41.759998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:41.759999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:41.760056 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:41.760263 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:41.760264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:41.760264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:41.760374 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:41.760427 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:41.760682 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 13:17:41.760768 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 13:17:41.761052 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:41.752468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35687 (local address 127.0.98.1:54277)
0925 13:17:41.752633 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:41.752670 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:41.753595 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:41.753803 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:41.753806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:41.754063 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:41.754266 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:41.754271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:41.755012 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:41.755013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:41.755847 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:41.755849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:41.756197 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:41.756198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:41.756298 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:41.756796 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:41.756804 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:41.758190 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:41.759610 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:41.759616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:41.759618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:41.759927 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:41.760105 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:41.760106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:41.760106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:41.760207 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:41.760425 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:41.760432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:41.760570 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 13:17:41.760659 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 13:17:41.760828 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260925 13:17:41.761490 4165 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35687
I20260925 13:17:41.761605 4165 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:41.761791 4165 heartbeater.cc:511] Master 127.0.98.62:35687 requested a full tablet report, sending...
I20260925 13:17:41.762342 3974 ts_manager.cc:194] Registered new tserver with Master: 5b3c8b2f04b640d0a9496b0e675cbf5a (127.0.98.1:46773)
I20260925 13:17:41.762929 3974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54277
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:17:41.783627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35687
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:41.864430 4170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:41.864637 4170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:41.864660 4170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:41.865850 4170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:41.865890 4170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:41.865916 4170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:17:41.867411 4170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:41.867602 4170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:41.867815 4170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:41.872625 4175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:41.872596 4178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:41.872596 4176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:41.872786 4170 server_base.cc:1034] running on GCE node
I20260925 13:17:41.873100 4170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:41.873344 4170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:41.874497 4170 hybrid_clock.cc:648] HybridClock initialized: now 1790342261874469 us; error 46 us; skew 500 ppm
Sep 25 13:17:41 dist-test-slave-t8d9 krb5kdc[3929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:41.876966 4170 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:17:41.877432 4170 webserver.cc:460] Webserver started at http://127.0.98.2:35787/ using document root <none> and password file <none>
I20260925 13:17:41.877664 4170 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:41.877709 4170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:41.877827 4170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:41.878897 4170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "8ed15f18aaf74e9f93c9febca5a58b08"
format_stamp: "Formatted at 2026-09-25 13:17:41 on dist-test-slave-t8d9"
server_key: "a87c8f7a1b2e30da6d6315d912b15062"
server_key_iv: "38f9d6b58bae43e8f99c71667bbd72e1"
server_key_version: "encryptionkey@0"
I20260925 13:17:41.879246 4170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8ed15f18aaf74e9f93c9febca5a58b08"
format_stamp: "Formatted at 2026-09-25 13:17:41 on dist-test-slave-t8d9"
server_key: "a87c8f7a1b2e30da6d6315d912b15062"
server_key_iv: "38f9d6b58bae43e8f99c71667bbd72e1"
server_key_version: "encryptionkey@0"
I20260925 13:17:41.880506 4170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:41.881204 4185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:41.881386 4170 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:41.881465 4170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "8ed15f18aaf74e9f93c9febca5a58b08"
format_stamp: "Formatted at 2026-09-25 13:17:41 on dist-test-slave-t8d9"
server_key: "a87c8f7a1b2e30da6d6315d912b15062"
server_key_iv: "38f9d6b58bae43e8f99c71667bbd72e1"
server_key_version: "encryptionkey@0"
I20260925 13:17:41.881554 4170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:41.894464 4170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:41.896454 4170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:41.896575 4170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:41.896806 4170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:41.897135 4170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:41.897168 4170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:41.897192 4170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:41.897214 4170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:41.902577 4170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32787
I20260925 13:17:41.902644 4298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32787 every 8 connection(s)
I20260925 13:17:41.902935 4170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:17:41.908401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 4170
I20260925 13:17:41.908501 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:17:41.908689 392 external_mini_cluster.cc:1428] Setting key 8256a55031041af047493ff3389b7a48
Sep 25 13:17:41 dist-test-slave-t8d9 krb5kdc[3929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:41.912153 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:41.903615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38181 (local address 127.0.98.62:35687)
0925 13:17:41.903695 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:41.903697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:41.903842 (+ 145us) server_negotiation.cc:407] Connection header received
0925 13:17:41.904787 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:41.904789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:41.904801 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:41.904853 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:41.905434 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:41.905985 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:41.906933 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:41.907128 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:41.907191 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:41.909343 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:41.909344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:41.909344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:41.909350 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:41.910682 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:41.911150 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:41.911151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:41.911152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:41.911203 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:41.911482 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:41.911484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:41.911484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:41.911607 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:41.911675 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:41.911974 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 13:17:41.912063 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 13:17:41.912231 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:41.903587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35687 (local address 127.0.98.2:38181)
0925 13:17:41.903727 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:41.903763 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:41.904656 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:41.904911 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:41.904914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:41.905163 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:41.905355 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:41.905359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:41.906053 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:41.906054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:41.906845 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:41.906847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:41.907196 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:41.907196 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:41.907290 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:41.907704 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:41.907712 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:41.909249 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:41.910775 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:41.910783 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:41.910786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:41.911077 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:41.911253 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:41.911253 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:41.911254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:41.911394 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:41.911673 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:41.911678 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:41.911833 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 13:17:41.911934 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 13:17:41.912058 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260925 13:17:41.912676 4299 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35687
I20260925 13:17:41.912781 4299 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:41.912968 4299 heartbeater.cc:511] Master 127.0.98.62:35687 requested a full tablet report, sending...
I20260925 13:17:41.913384 3974 ts_manager.cc:194] Registered new tserver with Master: 8ed15f18aaf74e9f93c9febca5a58b08 (127.0.98.2:32787)
I20260925 13:17:41.913837 3974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38181
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:17:41.932492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35687
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:42.013285 4304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:42.013521 4304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:42.013577 4304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:42.014807 4304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:42.014855 4304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:42.014892 4304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:17:42.016404 4304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:42.016649 4304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:42.016868 4304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:42.021899 4310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:42.021916 4304 server_base.cc:1034] running on GCE node
W20260925 13:17:42.021906 4312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:42.021932 4309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:42.022324 4304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:42.022539 4304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:42.023698 4304 hybrid_clock.cc:648] HybridClock initialized: now 1790342262023674 us; error 38 us; skew 500 ppm
Sep 25 13:17:42 dist-test-slave-t8d9 krb5kdc[3929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:42.026275 4304 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:17:42.026744 4304 webserver.cc:460] Webserver started at http://127.0.98.3:33437/ using document root <none> and password file <none>
I20260925 13:17:42.026969 4304 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:42.027029 4304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:42.027154 4304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:42.028209 4304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "f9a0f2f2881e4086b27cf386471cfda2"
format_stamp: "Formatted at 2026-09-25 13:17:42 on dist-test-slave-t8d9"
server_key: "8993d376ddeb13763630fa9d31d34043"
server_key_iv: "bc0712ee9257ea728aae5ab310597e47"
server_key_version: "encryptionkey@0"
I20260925 13:17:42.028558 4304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f9a0f2f2881e4086b27cf386471cfda2"
format_stamp: "Formatted at 2026-09-25 13:17:42 on dist-test-slave-t8d9"
server_key: "8993d376ddeb13763630fa9d31d34043"
server_key_iv: "bc0712ee9257ea728aae5ab310597e47"
server_key_version: "encryptionkey@0"
I20260925 13:17:42.029978 4304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:17:42.030831 4319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:42.031029 4304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:17:42.031118 4304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "f9a0f2f2881e4086b27cf386471cfda2"
format_stamp: "Formatted at 2026-09-25 13:17:42 on dist-test-slave-t8d9"
server_key: "8993d376ddeb13763630fa9d31d34043"
server_key_iv: "bc0712ee9257ea728aae5ab310597e47"
server_key_version: "encryptionkey@0"
I20260925 13:17:42.031184 4304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:42.069604 4304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:42.071478 4304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:42.071604 4304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:42.071839 4304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:42.072134 4304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:42.072165 4304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:42.072187 4304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:42.072209 4304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:42.077512 4304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38123
I20260925 13:17:42.077612 4432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38123 every 8 connection(s)
I20260925 13:17:42.077908 4304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:17:42 dist-test-slave-t8d9 krb5kdc[3929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:42.087255 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:42.078581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53087 (local address 127.0.98.62:35687)
0925 13:17:42.078647 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:42.078649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:42.078857 (+ 208us) server_negotiation.cc:407] Connection header received
0925 13:17:42.079784 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:42.079785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:42.079794 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:42.079841 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:42.080446 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:42.080947 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:42.082085 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:42.082269 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:42.082336 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:42.084471 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:42.084472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:42.084473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:42.084481 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:42.085991 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:42.086467 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:42.086468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:42.086468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:42.086524 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:42.086740 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:42.086740 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:42.086740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:42.086842 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:42.086890 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:42.087113 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 13:17:42.087186 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 13:17:42.087538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 4304
I20260925 13:17:42.087636 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:17:42.087811 392 external_mini_cluster.cc:1428] Setting key a3b9f95cf7c1395c1c1ad0b71bf96a69
I20260925 13:17:42.088042 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:42.078573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35687 (local address 127.0.98.3:53087)
0925 13:17:42.078727 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:42.078771 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:42.079674 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:42.079906 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:42.079909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:42.080172 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:42.080378 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:42.080383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:42.081021 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:42.081022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:42.082025 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:42.082028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:42.082337 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:42.082337 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:42.082432 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:42.082904 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:42.082930 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:42.084369 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:42.086094 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:42.086102 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:42.086104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:42.086389 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:42.086579 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:42.086580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:42.086580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:42.086693 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:42.086886 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:42.086890 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:42.087014 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 13:17:42.087096 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 13:17:42.087223 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260925 13:17:42.088459 4433 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35687
I20260925 13:17:42.088557 4433 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:42.088755 4433 heartbeater.cc:511] Master 127.0.98.62:35687 requested a full tablet report, sending...
I20260925 13:17:42.089124 3974 ts_manager.cc:194] Registered new tserver with Master: f9a0f2f2881e4086b27cf386471cfda2 (127.0.98.3:38123)
I20260925 13:17:42.089493 3974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53087
I20260925 13:17:42.099526 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 13:17:42 dist-test-slave-t8d9 krb5kdc[3929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 13:17:42 dist-test-slave-t8d9 krb5kdc[3929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:17:42.122118 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:42.113949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.0.98.1:46773)
0925 13:17:42.114127 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:42.114132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:42.114143 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:17:42.114170 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:42.114172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:42.114186 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:42.114253 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:42.114786 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:42.115538 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:42.116355 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:42.116713 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:42.116785 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:42.119116 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:42.119131 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:42.119131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:42.119139 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:42.120744 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:42.121153 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:42.121157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:42.121157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:42.121250 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:42.121467 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:42.121468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:42.121468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:42.121672 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:42.121818 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:42.121938 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 13:17:42.122013 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":100,"threads_started":1}
Sep 25 13:17:42 dist-test-slave-t8d9 krb5kdc[3929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:42.129998 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:42.123540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.0.98.62:35687)
0925 13:17:42.123587 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:42.123589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:42.123613 (+ 24us) server_negotiation.cc:407] Connection header received
0925 13:17:42.123677 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:42.123678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:42.123688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:42.123752 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:42.124259 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:42.124731 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:42.125471 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:42.125720 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:42.125785 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:42.127896 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:42.127898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:42.127898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:42.127905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:42.128976 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:42.129287 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:42.129288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:42.129289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:42.129332 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:42.129460 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:42.129460 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:42.129461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:42.129662 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:42.129744 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:42.129867 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 13:17:42.129929 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 13:17:42.130224 3974 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57770
I20260925 13:17:42.130522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 4036
I20260925 13:17:42.134500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 4170
I20260925 13:17:42.138681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 4304
I20260925 13:17:42.142220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 3942
2026-09-25T13:17:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1748 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:17:42 dist-test-slave-t8d9 krb5kdc[4445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:17:42 dist-test-slave-t8d9 krb5kdc[4445](info): set up 2 sockets
Sep 25 13:17:42 dist-test-slave-t8d9 krb5kdc[4445](info): commencing operation
krb5kdc: starting...
W20260925 13:17:43.093245 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:17:43 dist-test-slave-t8d9 krb5kdc[4445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:17:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:17:43.207528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43019
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:43.289683 4458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:43.289945 4458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:43.289983 4458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:43.291245 4458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:43.291294 4458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:43.291311 4458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:43.291327 4458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:43.291342 4458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:43.293002 4458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:43.293292 4458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:43.293511 4458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:17:43.298247 4458 server_base.cc:1034] running on GCE node
W20260925 13:17:43.298230 4466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:43.298249 4463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:43.298221 4464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:43.298666 4458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:43.298985 4458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:43.300149 4458 hybrid_clock.cc:648] HybridClock initialized: now 1790342263300129 us; error 33 us; skew 500 ppm
Sep 25 13:17:43 dist-test-slave-t8d9 krb5kdc[4445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:43.302630 4458 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:17:43.303001 4458 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:43.303057 4458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:43.303184 4458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:43.304262 4458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "2f6d5c5f108d4146867fd84fc8c221f9"
format_stamp: "Formatted at 2026-09-25 13:17:43 on dist-test-slave-t8d9"
server_key: "374c784b294ce388995c8b20d0d500c5"
server_key_iv: "f2770a1998b21e96cb32d0444013e00d"
server_key_version: "encryptionkey@0"
I20260925 13:17:43.304585 4458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "2f6d5c5f108d4146867fd84fc8c221f9"
format_stamp: "Formatted at 2026-09-25 13:17:43 on dist-test-slave-t8d9"
server_key: "374c784b294ce388995c8b20d0d500c5"
server_key_iv: "f2770a1998b21e96cb32d0444013e00d"
server_key_version: "encryptionkey@0"
I20260925 13:17:43.305833 4458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:43.306473 4472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:43.306614 4458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:43.306694 4458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "2f6d5c5f108d4146867fd84fc8c221f9"
format_stamp: "Formatted at 2026-09-25 13:17:43 on dist-test-slave-t8d9"
server_key: "374c784b294ce388995c8b20d0d500c5"
server_key_iv: "f2770a1998b21e96cb32d0444013e00d"
server_key_version: "encryptionkey@0"
I20260925 13:17:43.306752 4458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:43.334172 4458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:43.336319 4458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:43.336465 4458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:43.340409 4458 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43019
I20260925 13:17:43.340452 4524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43019 every 8 connection(s)
I20260925 13:17:43.340739 4458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:43.341746 4525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:43.342592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 4458
I20260925 13:17:43.342674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:43.342833 392 external_mini_cluster.cc:1428] Setting key 1d6652610366c9a2b376a10afaff2aef
I20260925 13:17:43.344904 4525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9: Bootstrap starting.
I20260925 13:17:43.345695 4525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:43.345995 4525 log.cc:824] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:43.346678 4525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9: No bootstrap required, opened a new log
I20260925 13:17:43.348462 4525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6d5c5f108d4146867fd84fc8c221f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260925 13:17:43.348609 4525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:43.348630 4525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6d5c5f108d4146867fd84fc8c221f9, State: Initialized, Role: FOLLOWER
I20260925 13:17:43.348747 4525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6d5c5f108d4146867fd84fc8c221f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260925 13:17:43.348820 4525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:43.348847 4525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:43.348893 4525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 13:17:43 dist-test-slave-t8d9 krb5kdc[4445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:43.349459 4525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6d5c5f108d4146867fd84fc8c221f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260925 13:17:43.349584 4525 leader_election.cc:302] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f6d5c5f108d4146867fd84fc8c221f9; no voters:
I20260925 13:17:43.349737 4525 leader_election.cc:288] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:43.349799 4530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:43.349994 4525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:43.350008 4530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f6d5c5f108d4146867fd84fc8c221f9, State: Running, Role: LEADER
I20260925 13:17:43.350421 4530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6d5c5f108d4146867fd84fc8c221f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260925 13:17:43.351191 4532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f6d5c5f108d4146867fd84fc8c221f9. Latest consensus state: current_term: 1 leader_uuid: "2f6d5c5f108d4146867fd84fc8c221f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6d5c5f108d4146867fd84fc8c221f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } } }
I20260925 13:17:43.351290 4532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:43.351480 4531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f6d5c5f108d4146867fd84fc8c221f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6d5c5f108d4146867fd84fc8c221f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } } }
I20260925 13:17:43.352360 4531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9 [sys.catalog]: This master's current role is: LEADER
W20260925 13:17:43.351737 4544 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:17:43.352418 4544 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:17:43.352617 4547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:43.352811 4547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:43.353492 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.343380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.0.98.62:43019)
0925 13:17:43.343584 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:43.343594 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:43.343630 (+ 36us) server_negotiation.cc:407] Connection header received
0925 13:17:43.344130 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:43.344141 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:43.344223 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:43.344384 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:43.344954 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.346299 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.347092 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.347508 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.347603 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.349859 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:43.349863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:43.349864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:43.349878 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:43.351848 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:43.352225 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:43.352227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:43.352227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:43.352331 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:43.352544 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:43.352545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:43.352546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:43.352866 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:43.353028 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:43.353138 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 13:17:43.353276 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260925 13:17:43.354636 4547 catalog_manager.cc:1347] Generated new cluster ID: 9ca65e9dc84b48aa90482e8a4a23ca9a
I20260925 13:17:43.354691 4547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:17:43.368062 4547 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:43.368600 4547 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:43.371187 4547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9: Generated new TSK 0
I20260925 13:17:43.371393 4547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:17:43.396682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:43.479001 4551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:43.479220 4551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:43.479255 4551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:43.480458 4551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:43.480502 4551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:43.480538 4551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:17:43.482103 4551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:43.482354 4551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:43.482564 4551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:43.487372 4557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:43.487376 4559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:43.487391 4556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:43.487392 4551 server_base.cc:1034] running on GCE node
I20260925 13:17:43.487833 4551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:43.488058 4551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:43.489229 4551 hybrid_clock.cc:648] HybridClock initialized: now 1790342263489216 us; error 32 us; skew 500 ppm
Sep 25 13:17:43 dist-test-slave-t8d9 krb5kdc[4445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:43.491909 4551 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:17:43.492272 4551 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:43.492323 4551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:43.492416 4551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:43.493435 4551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "fa843553f00f4fbea4290f3d2fcf7c5d"
format_stamp: "Formatted at 2026-09-25 13:17:43 on dist-test-slave-t8d9"
server_key: "63e4bf5ff59ace8062864b180799b15d"
server_key_iv: "e9ffb574eab6c32b301fbc567034cab7"
server_key_version: "encryptionkey@0"
I20260925 13:17:43.493799 4551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fa843553f00f4fbea4290f3d2fcf7c5d"
format_stamp: "Formatted at 2026-09-25 13:17:43 on dist-test-slave-t8d9"
server_key: "63e4bf5ff59ace8062864b180799b15d"
server_key_iv: "e9ffb574eab6c32b301fbc567034cab7"
server_key_version: "encryptionkey@0"
I20260925 13:17:43.495002 4551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:43.495646 4565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:43.495823 4551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:43.495903 4551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "fa843553f00f4fbea4290f3d2fcf7c5d"
format_stamp: "Formatted at 2026-09-25 13:17:43 on dist-test-slave-t8d9"
server_key: "63e4bf5ff59ace8062864b180799b15d"
server_key_iv: "e9ffb574eab6c32b301fbc567034cab7"
server_key_version: "encryptionkey@0"
I20260925 13:17:43.495970 4551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:43.508882 4551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:43.511149 4551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:43.511274 4551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:43.511503 4551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:43.511857 4551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:43.511894 4551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:43.511917 4551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:43.511941 4551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:43.517462 4551 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41963
I20260925 13:17:43.517508 4678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41963 every 8 connection(s)
I20260925 13:17:43.517840 4551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:17:43.521993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 4551
I20260925 13:17:43.522105 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:17:43.522280 392 external_mini_cluster.cc:1428] Setting key 49ce9575dfb0e4aa48ac61322db39b77
Sep 25 13:17:43 dist-test-slave-t8d9 krb5kdc[4445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:43.527343 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.518543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35473 (local address 127.0.98.62:43019)
0925 13:17:43.518605 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:43.518607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:43.518779 (+ 172us) server_negotiation.cc:407] Connection header received
0925 13:17:43.519250 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:43.519251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:43.519261 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:43.519329 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:43.520214 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.520708 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.521785 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.522087 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.522159 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.524460 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:43.524461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:43.524462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:43.524468 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:43.525806 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:43.526317 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:43.526318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:43.526318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:43.526376 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:43.526646 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:43.526646 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:43.526647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:43.526748 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:43.526792 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:43.527167 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 13:17:43.527271 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 13:17:43.527544 4681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.518509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:35473)
0925 13:17:43.518657 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:43.518700 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:43.519174 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:43.519389 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:43.519393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:43.519655 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:43.520140 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.520146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.520826 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.520826 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.521698 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.521701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.522191 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.522192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.522300 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.522846 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:43.522863 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:43.524360 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:43.525910 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:43.525917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:43.525920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:43.526219 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:43.526442 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:43.526443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:43.526444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:43.526582 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:43.526805 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:43.526811 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:43.526993 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 13:17:43.527116 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 13:17:43.527296 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260925 13:17:43.528044 4679 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43019
I20260925 13:17:43.528147 4679 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:43.528321 4679 heartbeater.cc:511] Master 127.0.98.62:43019 requested a full tablet report, sending...
I20260925 13:17:43.528878 4489 ts_manager.cc:194] Registered new tserver with Master: fa843553f00f4fbea4290f3d2fcf7c5d (127.0.98.1:41963)
I20260925 13:17:43.529580 4489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35473
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:17:43.545002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:43.626159 4684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:43.626379 4684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:43.626410 4684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:43.627580 4684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:43.627624 4684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:43.627660 4684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:17:43.629146 4684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:43.629380 4684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:43.629606 4684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:43.634369 4692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:43.634455 4684 server_base.cc:1034] running on GCE node
W20260925 13:17:43.634380 4689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:43.634366 4690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:43.634774 4684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:43.635012 4684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:43.636149 4684 hybrid_clock.cc:648] HybridClock initialized: now 1790342263636137 us; error 31 us; skew 500 ppm
Sep 25 13:17:43 dist-test-slave-t8d9 krb5kdc[4445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:43.638600 4684 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:17:43.638903 4684 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:43.638952 4684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:43.639075 4684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:43.640219 4684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "4068e43cc351483781e04af2540f656c"
format_stamp: "Formatted at 2026-09-25 13:17:43 on dist-test-slave-t8d9"
server_key: "807ea0c0d241e45cff436625ee69b511"
server_key_iv: "482dfd76255f5efca727694f64ea55df"
server_key_version: "encryptionkey@0"
I20260925 13:17:43.640563 4684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4068e43cc351483781e04af2540f656c"
format_stamp: "Formatted at 2026-09-25 13:17:43 on dist-test-slave-t8d9"
server_key: "807ea0c0d241e45cff436625ee69b511"
server_key_iv: "482dfd76255f5efca727694f64ea55df"
server_key_version: "encryptionkey@0"
I20260925 13:17:43.641885 4684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:17:43.642655 4698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:43.642823 4684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:17:43.642916 4684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "4068e43cc351483781e04af2540f656c"
format_stamp: "Formatted at 2026-09-25 13:17:43 on dist-test-slave-t8d9"
server_key: "807ea0c0d241e45cff436625ee69b511"
server_key_iv: "482dfd76255f5efca727694f64ea55df"
server_key_version: "encryptionkey@0"
I20260925 13:17:43.642977 4684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:43.656400 4684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:43.658644 4684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:43.658789 4684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:43.659014 4684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:43.659329 4684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:43.659363 4684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:43.659390 4684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:43.659407 4684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:43.664880 4684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35171
I20260925 13:17:43.664950 4811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35171 every 8 connection(s)
I20260925 13:17:43.665182 4684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:17:43.669745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 4684
I20260925 13:17:43.669855 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:17:43.670032 392 external_mini_cluster.cc:1428] Setting key aa548aeaf86bce76d5694c0fc4439f3b
Sep 25 13:17:43 dist-test-slave-t8d9 krb5kdc[4445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:43.674595 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.665864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59867 (local address 127.0.98.62:43019)
0925 13:17:43.665944 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:43.665947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:43.666144 (+ 197us) server_negotiation.cc:407] Connection header received
0925 13:17:43.666595 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:43.666596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:43.666606 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:43.666654 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:43.667589 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.668094 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.669117 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.669314 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.669369 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.671786 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:43.671788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:43.671788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:43.671796 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:43.673139 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:43.673625 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:43.673627 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:43.673627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:43.673688 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:43.673962 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:43.673963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:43.673963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:43.674080 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:43.674135 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:43.674460 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 13:17:43.674528 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 13:17:43.674866 4814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.665864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:59867)
0925 13:17:43.666024 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:43.666062 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:43.666520 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:43.666711 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:43.666715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:43.667019 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:43.667506 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.667513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.668176 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.668177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.669030 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.669032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.669376 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.669377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.669468 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.670080 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:43.670089 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:43.671700 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:43.673210 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:43.673217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:43.673220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:43.673493 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:43.673750 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:43.673751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:43.673751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:43.673896 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:43.674130 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:43.674137 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:43.674319 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 13:17:43.674435 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 13:17:43.674611 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260925 13:17:43.675351 4812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43019
I20260925 13:17:43.675452 4812 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:43.675611 4812 heartbeater.cc:511] Master 127.0.98.62:43019 requested a full tablet report, sending...
I20260925 13:17:43.676002 4489 ts_manager.cc:194] Registered new tserver with Master: 4068e43cc351483781e04af2540f656c (127.0.98.2:35171)
I20260925 13:17:43.676494 4489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59867
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:17:43.694033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:43.773716 4817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:43.773931 4817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:43.773963 4817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:43.775134 4817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:43.775174 4817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:43.775210 4817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:17:43.776703 4817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:43.776944 4817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:43.777166 4817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:43.781852 4825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:43.781912 4822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:43.781908 4823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:43.782092 4817 server_base.cc:1034] running on GCE node
I20260925 13:17:43.782317 4817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:43.782558 4817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:43.783720 4817 hybrid_clock.cc:648] HybridClock initialized: now 1790342263783702 us; error 25 us; skew 500 ppm
Sep 25 13:17:43 dist-test-slave-t8d9 krb5kdc[4445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:43.786279 4817 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:17:43.786638 4817 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:43.786692 4817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:43.786814 4817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:43.787912 4817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "23291034fa1a444cbf7d2978b3ebf96c"
format_stamp: "Formatted at 2026-09-25 13:17:43 on dist-test-slave-t8d9"
server_key: "280dcae6a5ae8929753b90cd4ee9b849"
server_key_iv: "bcaa4f8e110c3167ae172cb1ecdfdf72"
server_key_version: "encryptionkey@0"
I20260925 13:17:43.788237 4817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "23291034fa1a444cbf7d2978b3ebf96c"
format_stamp: "Formatted at 2026-09-25 13:17:43 on dist-test-slave-t8d9"
server_key: "280dcae6a5ae8929753b90cd4ee9b849"
server_key_iv: "bcaa4f8e110c3167ae172cb1ecdfdf72"
server_key_version: "encryptionkey@0"
I20260925 13:17:43.789577 4817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:17:43.790397 4831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:43.790555 4817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:17:43.790647 4817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "23291034fa1a444cbf7d2978b3ebf96c"
format_stamp: "Formatted at 2026-09-25 13:17:43 on dist-test-slave-t8d9"
server_key: "280dcae6a5ae8929753b90cd4ee9b849"
server_key_iv: "bcaa4f8e110c3167ae172cb1ecdfdf72"
server_key_version: "encryptionkey@0"
I20260925 13:17:43.790719 4817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:43.798292 4817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:43.800377 4817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:43.800513 4817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:43.800730 4817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:43.801023 4817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:43.801054 4817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:43.801083 4817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:43.801107 4817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:43.806702 4817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35869
I20260925 13:17:43.806743 4944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35869 every 8 connection(s)
I20260925 13:17:43.807010 4817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:17:43.808593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 4817
I20260925 13:17:43.808720 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:17:43.808966 392 external_mini_cluster.cc:1428] Setting key 0227e0cc8f84a3035f11bae764c39263
Sep 25 13:17:43 dist-test-slave-t8d9 krb5kdc[4445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:43.816138 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.807715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47869 (local address 127.0.98.62:43019)
0925 13:17:43.807761 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:43.807763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:43.807947 (+ 184us) server_negotiation.cc:407] Connection header received
0925 13:17:43.808466 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:43.808467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:43.808480 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:43.808532 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:43.809672 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.810231 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.811225 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.811421 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.811475 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.813576 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:43.813578 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:43.813578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:43.813586 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:43.814866 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:43.815252 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:43.815253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:43.815254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:43.815311 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:43.815527 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:43.815528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:43.815528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:43.815642 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:43.815696 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:43.815968 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 13:17:43.816045 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:17:43.816347 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.807686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:47869)
0925 13:17:43.807830 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:43.807866 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:43.808367 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:43.808603 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:43.808607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:43.808936 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:43.809587 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.809595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.810312 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.810313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.811135 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.811137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.811490 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.811492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.811618 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.812070 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:43.812078 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:43.813459 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:43.814954 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:43.814962 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:43.814965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:43.815184 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:43.815364 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:43.815365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:43.815366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:43.815473 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:43.815697 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:43.815704 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:43.815850 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 13:17:43.815942 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 13:17:43.816108 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260925 13:17:43.816839 4945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43019
I20260925 13:17:43.816927 4945 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:43.817055 4945 heartbeater.cc:511] Master 127.0.98.62:43019 requested a full tablet report, sending...
I20260925 13:17:43.817368 4489 ts_manager.cc:194] Registered new tserver with Master: 23291034fa1a444cbf7d2978b3ebf96c (127.0.98.3:35869)
I20260925 13:17:43.817824 4489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47869
I20260925 13:17:43.820750 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:17:43.827625 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.822095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.0.98.62:43019)
0925 13:17:43.822155 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:43.822157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:43.822241 (+ 84us) server_negotiation.cc:407] Connection header received
0925 13:17:43.822310 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:43.822311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:43.822321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:43.822371 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:43.822931 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.823411 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.824128 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.824311 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.824365 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.825490 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:43.825491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:43.825491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:43.825499 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:43.826557 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:43.826889 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:43.826890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:43.826891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:43.826942 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:43.827097 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:43.827098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:43.827098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:43.827250 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:43.827339 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:43.827461 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 13:17:43.827535 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:17:43.828949 4489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:17:43.830484 4489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:43.836583 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.832862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58860 (local address 127.0.98.1:41963)
0925 13:17:43.833071 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:43.833076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:43.833086 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:17:43.833124 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:43.833126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:43.833143 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:43.833240 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:43.833387 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.834537 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.835718 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.836207 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.836270 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.836330 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:43.836460 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 13:17:43.836501 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":108,"threads_started":1}
I20260925 13:17:43.836570 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.832577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41963 (local address 127.0.0.1:58860)
0925 13:17:43.832783 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:43.832831 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:43.832898 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:43.833244 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:43.833247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:43.833249 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:43.833317 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.833319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.834632 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.834633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.835650 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.835652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.836288 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.836289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.836364 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.836376 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 13:17:43.836440 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 13:17:43.836492 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":80,"threads_started":1}
I20260925 13:17:43.836704 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.832578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35171 (local address 127.0.0.1:36066)
0925 13:17:43.832992 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:43.833000 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:43.833046 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:43.833290 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:43.833291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:43.833293 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:43.833339 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.833341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.834284 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.834285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.835449 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.835451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.836389 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.836390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.836451 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.836455 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:43.836514 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 13:17:43.836556 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":89,"threads_started":1}
I20260925 13:17:43.836741 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.832850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36066 (local address 127.0.98.2:35171)
0925 13:17:43.832979 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:43.832984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:43.833040 (+ 56us) server_negotiation.cc:407] Connection header received
0925 13:17:43.833104 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:43.833105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:43.833117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:43.833191 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:43.833411 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.834203 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.835527 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.836240 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.836305 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.836355 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:43.836614 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 13:17:43.836655 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
I20260925 13:17:43.837188 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.832579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35869 (local address 127.0.0.1:44788)
0925 13:17:43.833116 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:43.833123 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:43.833166 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:43.833565 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:43.833565 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:43.833568 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:43.833605 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.833606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.834774 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.834775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.835649 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.835651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.836531 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.836532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.836579 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.836582 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:43.837078 (+ 496us) client_negotiation.cc:240] Negotiation successful
0925 13:17:43.837123 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":4,"spinlock_wait_cycles":43392,"thread_start_us":98,"threads_started":1}
I20260925 13:17:43.837420 4613 tablet_service.cc:1467] Processing CreateTablet for tablet 85755f8a0ee044b3b01d12c5ee2c5c3f (DEFAULT_TABLE table=test-table [id=c2d2cdc86ef34726a38aa7e3c8287f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:43.837445 4746 tablet_service.cc:1467] Processing CreateTablet for tablet 85755f8a0ee044b3b01d12c5ee2c5c3f (DEFAULT_TABLE table=test-table [id=c2d2cdc86ef34726a38aa7e3c8287f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:43.837613 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.832744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44788 (local address 127.0.98.3:35869)
0925 13:17:43.833010 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:43.833013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:43.833178 (+ 165us) server_negotiation.cc:407] Connection header received
0925 13:17:43.833380 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:43.833383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:43.833400 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:43.833465 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:43.833681 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.834696 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.835733 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.836387 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.836452 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.836501 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:43.837472 (+ 971us) server_negotiation.cc:299] Negotiation successful
0925 13:17:43.837517 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":83,"threads_started":1}
I20260925 13:17:43.837814 4613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85755f8a0ee044b3b01d12c5ee2c5c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:43.837814 4746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85755f8a0ee044b3b01d12c5ee2c5c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:43.840301 4959 tablet_bootstrap.cc:492] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c: Bootstrap starting.
I20260925 13:17:43.840471 4960 tablet_bootstrap.cc:492] T 85755f8a0ee044b3b01d12c5ee2c5c3f P fa843553f00f4fbea4290f3d2fcf7c5d: Bootstrap starting.
I20260925 13:17:43.840968 4959 tablet_bootstrap.cc:654] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:43.840835 4879 tablet_service.cc:1467] Processing CreateTablet for tablet 85755f8a0ee044b3b01d12c5ee2c5c3f (DEFAULT_TABLE table=test-table [id=c2d2cdc86ef34726a38aa7e3c8287f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:43.840972 4960 tablet_bootstrap.cc:654] T 85755f8a0ee044b3b01d12c5ee2c5c3f P fa843553f00f4fbea4290f3d2fcf7c5d: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:43.841182 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85755f8a0ee044b3b01d12c5ee2c5c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:43.841492 4960 log.cc:824] T 85755f8a0ee044b3b01d12c5ee2c5c3f P fa843553f00f4fbea4290f3d2fcf7c5d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:43.841526 4959 log.cc:824] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:43.842269 4960 tablet_bootstrap.cc:492] T 85755f8a0ee044b3b01d12c5ee2c5c3f P fa843553f00f4fbea4290f3d2fcf7c5d: No bootstrap required, opened a new log
I20260925 13:17:43.842341 4960 ts_tablet_manager.cc:1397] T 85755f8a0ee044b3b01d12c5ee2c5c3f P fa843553f00f4fbea4290f3d2fcf7c5d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 13:17:43.844308 4963 tablet_bootstrap.cc:492] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c: Bootstrap starting.
I20260925 13:17:43.844275 4960 raft_consensus.cc:348] T 85755f8a0ee044b3b01d12c5ee2c5c3f P fa843553f00f4fbea4290f3d2fcf7c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23291034fa1a444cbf7d2978b3ebf96c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35869 } } peers { permanent_uuid: "4068e43cc351483781e04af2540f656c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35171 } } peers { permanent_uuid: "fa843553f00f4fbea4290f3d2fcf7c5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41963 } }
I20260925 13:17:43.844463 4960 raft_consensus.cc:374] T 85755f8a0ee044b3b01d12c5ee2c5c3f P fa843553f00f4fbea4290f3d2fcf7c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:43.844507 4960 raft_consensus.cc:729] T 85755f8a0ee044b3b01d12c5ee2c5c3f P fa843553f00f4fbea4290f3d2fcf7c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa843553f00f4fbea4290f3d2fcf7c5d, State: Initialized, Role: FOLLOWER
I20260925 13:17:43.844615 4960 consensus_queue.cc:260] T 85755f8a0ee044b3b01d12c5ee2c5c3f P fa843553f00f4fbea4290f3d2fcf7c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23291034fa1a444cbf7d2978b3ebf96c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35869 } } peers { permanent_uuid: "4068e43cc351483781e04af2540f656c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35171 } } peers { permanent_uuid: "fa843553f00f4fbea4290f3d2fcf7c5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41963 } }
I20260925 13:17:43.844893 4960 ts_tablet_manager.cc:1428] T 85755f8a0ee044b3b01d12c5ee2c5c3f P fa843553f00f4fbea4290f3d2fcf7c5d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 13:17:43.844911 4963 tablet_bootstrap.cc:654] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:43.844944 4679 heartbeater.cc:503] Master 127.0.98.62:43019 was elected leader, sending a full tablet report...
I20260925 13:17:43.845299 4959 tablet_bootstrap.cc:492] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c: No bootstrap required, opened a new log
I20260925 13:17:43.845368 4959 ts_tablet_manager.cc:1397] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 13:17:43.845362 4963 log.cc:824] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:43.846115 4963 tablet_bootstrap.cc:492] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c: No bootstrap required, opened a new log
I20260925 13:17:43.846184 4963 ts_tablet_manager.cc:1397] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 13:17:43.847342 4959 raft_consensus.cc:348] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23291034fa1a444cbf7d2978b3ebf96c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35869 } } peers { permanent_uuid: "4068e43cc351483781e04af2540f656c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35171 } } peers { permanent_uuid: "fa843553f00f4fbea4290f3d2fcf7c5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41963 } }
I20260925 13:17:43.847580 4959 raft_consensus.cc:374] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:43.847626 4959 raft_consensus.cc:729] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4068e43cc351483781e04af2540f656c, State: Initialized, Role: FOLLOWER
I20260925 13:17:43.847738 4959 consensus_queue.cc:260] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23291034fa1a444cbf7d2978b3ebf96c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35869 } } peers { permanent_uuid: "4068e43cc351483781e04af2540f656c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35171 } } peers { permanent_uuid: "fa843553f00f4fbea4290f3d2fcf7c5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41963 } }
I20260925 13:17:43.848002 4812 heartbeater.cc:503] Master 127.0.98.62:43019 was elected leader, sending a full tablet report...
I20260925 13:17:43.848001 4959 ts_tablet_manager.cc:1428] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:17:43.848109 4963 raft_consensus.cc:348] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23291034fa1a444cbf7d2978b3ebf96c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35869 } } peers { permanent_uuid: "4068e43cc351483781e04af2540f656c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35171 } } peers { permanent_uuid: "fa843553f00f4fbea4290f3d2fcf7c5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41963 } }
I20260925 13:17:43.848237 4963 raft_consensus.cc:374] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:43.848265 4963 raft_consensus.cc:729] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23291034fa1a444cbf7d2978b3ebf96c, State: Initialized, Role: FOLLOWER
I20260925 13:17:43.848381 4963 consensus_queue.cc:260] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23291034fa1a444cbf7d2978b3ebf96c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35869 } } peers { permanent_uuid: "4068e43cc351483781e04af2540f656c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35171 } } peers { permanent_uuid: "fa843553f00f4fbea4290f3d2fcf7c5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41963 } }
I20260925 13:17:43.848609 4963 ts_tablet_manager.cc:1428] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 13:17:43.848716 4945 heartbeater.cc:503] Master 127.0.98.62:43019 was elected leader, sending a full tablet report...
I20260925 13:17:43.855201 4966 raft_consensus.cc:482] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:43.855342 4966 raft_consensus.cc:504] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23291034fa1a444cbf7d2978b3ebf96c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35869 } } peers { permanent_uuid: "4068e43cc351483781e04af2540f656c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35171 } } peers { permanent_uuid: "fa843553f00f4fbea4290f3d2fcf7c5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41963 } }
I20260925 13:17:43.855700 4966 leader_election.cc:288] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23291034fa1a444cbf7d2978b3ebf96c (127.0.98.3:35869), fa843553f00f4fbea4290f3d2fcf7c5d (127.0.98.1:41963)
I20260925 13:17:43.859320 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.855953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41417 (local address 127.0.98.1:41963)
0925 13:17:43.855995 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:43.855997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:43.856234 (+ 237us) server_negotiation.cc:407] Connection header received
0925 13:17:43.856281 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:43.856282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:43.856294 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:43.856372 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:43.856468 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.857137 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.858390 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.858964 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.859037 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.859047 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:43.859210 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 13:17:43.859257 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:17:43.859320 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.855938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41447 (local address 127.0.98.3:35869)
0925 13:17:43.856004 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:43.856005 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:43.856013 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:17:43.856036 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:43.856037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:43.856046 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:43.856088 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:43.856254 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.856995 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.858390 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.858965 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.859037 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.859047 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:43.859210 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 13:17:43.859248 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 13:17:43.859337 4814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.855852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35869 (local address 127.0.98.2:41447)
0925 13:17:43.855912 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:43.855923 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:43.855962 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:43.856100 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:43.856100 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:43.856102 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:43.856176 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.856178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.857070 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.857070 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.858282 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.858285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.859051 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.859052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.859155 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.859156 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 13:17:43.859200 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 13:17:43.859248 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":7}
I20260925 13:17:43.859686 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85755f8a0ee044b3b01d12c5ee2c5c3f" candidate_uuid: "4068e43cc351483781e04af2540f656c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23291034fa1a444cbf7d2978b3ebf96c" is_pre_election: true
I20260925 13:17:43.859349 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.855852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41963 (local address 127.0.98.2:41417)
0925 13:17:43.856163 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:43.856172 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:43.856220 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:43.856372 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:43.856373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:43.856374 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:43.856419 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.856420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.857192 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.857193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.858288 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:43.858290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.859051 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.859052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:43.859147 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.859151 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:43.859200 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 13:17:43.859247 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":23,"thread_start_us":83,"threads_started":1}
I20260925 13:17:43.859886 4899 raft_consensus.cc:2454] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4068e43cc351483781e04af2540f656c in term 0.
I20260925 13:17:43.860093 4633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85755f8a0ee044b3b01d12c5ee2c5c3f" candidate_uuid: "4068e43cc351483781e04af2540f656c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa843553f00f4fbea4290f3d2fcf7c5d" is_pre_election: true
I20260925 13:17:43.860132 4700 leader_election.cc:302] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23291034fa1a444cbf7d2978b3ebf96c, 4068e43cc351483781e04af2540f656c; no voters:
I20260925 13:17:43.860224 4633 raft_consensus.cc:2454] T 85755f8a0ee044b3b01d12c5ee2c5c3f P fa843553f00f4fbea4290f3d2fcf7c5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4068e43cc351483781e04af2540f656c in term 0.
I20260925 13:17:43.860270 4966 raft_consensus.cc:2781] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:43.860337 4966 raft_consensus.cc:482] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:43.860364 4966 raft_consensus.cc:3037] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:43.861102 4966 raft_consensus.cc:504] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23291034fa1a444cbf7d2978b3ebf96c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35869 } } peers { permanent_uuid: "4068e43cc351483781e04af2540f656c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35171 } } peers { permanent_uuid: "fa843553f00f4fbea4290f3d2fcf7c5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41963 } }
I20260925 13:17:43.861218 4966 leader_election.cc:288] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [CANDIDATE]: Term 1 election: Requested vote from peers 23291034fa1a444cbf7d2978b3ebf96c (127.0.98.3:35869), fa843553f00f4fbea4290f3d2fcf7c5d (127.0.98.1:41963)
I20260925 13:17:43.861379 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85755f8a0ee044b3b01d12c5ee2c5c3f" candidate_uuid: "4068e43cc351483781e04af2540f656c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23291034fa1a444cbf7d2978b3ebf96c"
I20260925 13:17:43.861395 4633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85755f8a0ee044b3b01d12c5ee2c5c3f" candidate_uuid: "4068e43cc351483781e04af2540f656c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa843553f00f4fbea4290f3d2fcf7c5d"
I20260925 13:17:43.861444 4899 raft_consensus.cc:3037] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:43.861459 4633 raft_consensus.cc:3037] T 85755f8a0ee044b3b01d12c5ee2c5c3f P fa843553f00f4fbea4290f3d2fcf7c5d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:43.862133 4899 raft_consensus.cc:2454] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4068e43cc351483781e04af2540f656c in term 1.
I20260925 13:17:43.862142 4633 raft_consensus.cc:2454] T 85755f8a0ee044b3b01d12c5ee2c5c3f P fa843553f00f4fbea4290f3d2fcf7c5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4068e43cc351483781e04af2540f656c in term 1.
I20260925 13:17:43.862295 4702 leader_election.cc:302] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4068e43cc351483781e04af2540f656c, fa843553f00f4fbea4290f3d2fcf7c5d; no voters:
I20260925 13:17:43.862396 4966 raft_consensus.cc:2781] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:43.862534 4966 raft_consensus.cc:686] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [term 1 LEADER]: Becoming Leader. State: Replica: 4068e43cc351483781e04af2540f656c, State: Running, Role: LEADER
I20260925 13:17:43.862615 4966 consensus_queue.cc:237] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23291034fa1a444cbf7d2978b3ebf96c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35869 } } peers { permanent_uuid: "4068e43cc351483781e04af2540f656c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35171 } } peers { permanent_uuid: "fa843553f00f4fbea4290f3d2fcf7c5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41963 } }
I20260925 13:17:43.863376 4489 catalog_manager.cc:5495] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4068e43cc351483781e04af2540f656c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4068e43cc351483781e04af2540f656c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23291034fa1a444cbf7d2978b3ebf96c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4068e43cc351483781e04af2540f656c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa843553f00f4fbea4290f3d2fcf7c5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41963 } health_report { overall_health: UNKNOWN } } }
I20260925 13:17:43.870916 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.868548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36072 (local address 127.0.98.2:35171)
0925 13:17:43.868589 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:43.868590 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:43.868633 (+ 43us) server_negotiation.cc:407] Connection header received
0925 13:17:43.868695 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:43.868696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:43.868711 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:43.868765 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:43.868866 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.869318 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.870176 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:43.870427 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:43.870487 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:43.870618 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:43.870720 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:43.870774 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:43.870804 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 13:17:43.870826 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:17:43.872717 4633 raft_consensus.cc:1261] T 85755f8a0ee044b3b01d12c5ee2c5c3f P fa843553f00f4fbea4290f3d2fcf7c5d [term 1 FOLLOWER]: Refusing update from remote peer 4068e43cc351483781e04af2540f656c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:43.872717 4899 raft_consensus.cc:1261] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c [term 1 FOLLOWER]: Refusing update from remote peer 4068e43cc351483781e04af2540f656c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:43.873070 4966 consensus_queue.cc:1035] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [LEADER]: Connected to new peer: Peer: permanent_uuid: "23291034fa1a444cbf7d2978b3ebf96c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:43.873200 4969 consensus_queue.cc:1035] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa843553f00f4fbea4290f3d2fcf7c5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:43.875960 4974 mvcc.cc:205] Tried to move back new op lower bound from 7333241912820105216 to 7333241912781959168. Current Snapshot: MvccSnapshot[applied={T|T < 7333241912820105216}]
I20260925 13:17:43.875905 4973 mvcc.cc:205] Tried to move back new op lower bound from 7333241912820105216 to 7333241912781959168. Current Snapshot: MvccSnapshot[applied={T|T < 7333241912820105216}]
I20260925 13:17:43.877740 4488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:43.877848 4488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:43.878664 4488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f6d5c5f108d4146867fd84fc8c221f9: Sending DeleteTablet for 3 replicas of tablet 85755f8a0ee044b3b01d12c5ee2c5c3f
I20260925 13:17:43.879010 4613 tablet_service.cc:1514] Processing DeleteTablet for tablet 85755f8a0ee044b3b01d12c5ee2c5c3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58860
I20260925 13:17:43.879002 4746 tablet_service.cc:1514] Processing DeleteTablet for tablet 85755f8a0ee044b3b01d12c5ee2c5c3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36066
I20260925 13:17:43.879016 4879 tablet_service.cc:1514] Processing DeleteTablet for tablet 85755f8a0ee044b3b01d12c5ee2c5c3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44788
I20260925 13:17:43.879240 4983 tablet_replica.cc:330] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c: stopping tablet replica
I20260925 13:17:43.879346 4983 raft_consensus.cc:2229] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:43.879467 4983 raft_consensus.cc:2258] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:43.879490 4985 tablet_replica.cc:330] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c: stopping tablet replica
I20260925 13:17:43.879637 4984 tablet_replica.cc:330] T 85755f8a0ee044b3b01d12c5ee2c5c3f P fa843553f00f4fbea4290f3d2fcf7c5d: stopping tablet replica
I20260925 13:17:43.879647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 4551
I20260925 13:17:43.879657 4985 raft_consensus.cc:2229] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:17:43.879851 4985 raft_consensus.cc:2258] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:43.880595 4985 ts_tablet_manager.cc:1905] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:43.880820 4983 ts_tablet_manager.cc:1905] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:43.882489 4985 ts_tablet_manager.cc:1918] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:43.882489 4983 ts_tablet_manager.cc:1918] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:43.882561 4983 log.cc:1196] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/85755f8a0ee044b3b01d12c5ee2c5c3f
I20260925 13:17:43.882678 4985 log.cc:1196] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/85755f8a0ee044b3b01d12c5ee2c5c3f
I20260925 13:17:43.882823 4983 ts_tablet_manager.cc:1939] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 23291034fa1a444cbf7d2978b3ebf96c: Deleting consensus metadata
I20260925 13:17:43.883035 4985 ts_tablet_manager.cc:1939] T 85755f8a0ee044b3b01d12c5ee2c5c3f P 4068e43cc351483781e04af2540f656c: Deleting consensus metadata
I20260925 13:17:43.883430 4474 catalog_manager.cc:4841] TS 23291034fa1a444cbf7d2978b3ebf96c (127.0.98.3:35869): tablet 85755f8a0ee044b3b01d12c5ee2c5c3f (table test-table [id=c2d2cdc86ef34726a38aa7e3c8287f4e]) successfully deleted
I20260925 13:17:43.883437 4473 catalog_manager.cc:4841] TS 4068e43cc351483781e04af2540f656c (127.0.98.2:35171): tablet 85755f8a0ee044b3b01d12c5ee2c5c3f (table test-table [id=c2d2cdc86ef34726a38aa7e3c8287f4e]) successfully deleted
W20260925 13:17:43.885151 4476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41963 (error 108)
I20260925 13:17:43.885638 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:43.885477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41963 (local address 127.0.0.1:58872)
0925 13:17:43.885563 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:43.885603 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 13:17:43.885829 4476 catalog_manager.cc:4584] TS fa843553f00f4fbea4290f3d2fcf7c5d (127.0.98.1:41963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85755f8a0ee044b3b01d12c5ee2c5c3f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41963: connect: Connection refused (error 111)
I20260925 13:17:43.886746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 4684
I20260925 13:17:43.891167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 4817
I20260925 13:17:43.895498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 4458
2026-09-25T13:17:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1753 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T13:17:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:43Z Disabled control of system clock
I20260925 13:17:43.920733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33721
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 13:17:43.999361 4990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:43.999585 4990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:43.999605 4990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:44.000734 4990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:44.000772 4990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:44.000782 4990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:44.000793 4990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:44.000804 4990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:44.002163 4990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:44.002394 4990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:44.002646 4990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:17:44.007570 4990 server_base.cc:1034] running on GCE node
W20260925 13:17:44.007550 4995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:44.007555 4996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:44.007706 4998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:44.007983 4990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:44.008302 4990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:44.009457 4990 hybrid_clock.cc:648] HybridClock initialized: now 1790342264009449 us; error 34 us; skew 500 ppm
I20260925 13:17:44.010749 4990 webserver.cc:460] Webserver started at http://127.0.98.62:34129/ using document root <none> and password file <none>
I20260925 13:17:44.010991 4990 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:44.011042 4990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:44.011174 4990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:44.012245 4990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "b74819e38db24656aaa2ed0a1afdef73"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "5d2a717ea630d4c64768efb248a7debf"
server_key_iv: "3d9604cd1aae1857a44ade0b568177e7"
server_key_version: "encryptionkey@0"
I20260925 13:17:44.012588 4990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "b74819e38db24656aaa2ed0a1afdef73"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "5d2a717ea630d4c64768efb248a7debf"
server_key_iv: "3d9604cd1aae1857a44ade0b568177e7"
server_key_version: "encryptionkey@0"
I20260925 13:17:44.013857 4990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:44.014624 5004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:44.014817 4990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:44.014907 4990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "b74819e38db24656aaa2ed0a1afdef73"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "5d2a717ea630d4c64768efb248a7debf"
server_key_iv: "3d9604cd1aae1857a44ade0b568177e7"
server_key_version: "encryptionkey@0"
I20260925 13:17:44.014966 4990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:44.018272 4990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:44.018505 4990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:44.018595 4990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:44.022075 4990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33721
I20260925 13:17:44.022113 5056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33721 every 8 connection(s)
I20260925 13:17:44.022416 4990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:44.023054 5057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:44.024554 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 4990
I20260925 13:17:44.024626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:44.024775 392 external_mini_cluster.cc:1428] Setting key 77005b548c1afeec6d42c598628df495
I20260925 13:17:44.026170 5057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73: Bootstrap starting.
I20260925 13:17:44.026840 5057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:44.027161 5057 log.cc:824] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:44.027338 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.025284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.0.98.62:33721)
0925 13:17:44.025492 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:44.025501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:44.025559 (+ 58us) server_negotiation.cc:407] Connection header received
0925 13:17:44.026047 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:44.026056 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:44.026109 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:44.026261 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:44.026810 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:44.026815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:44.026815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:44.026819 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:44.026865 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:44.026888 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:44.026972 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:44.027087 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 13:17:44.027162 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260925 13:17:44.027884 5057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73: No bootstrap required, opened a new log
I20260925 13:17:44.029197 5057 raft_consensus.cc:348] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74819e38db24656aaa2ed0a1afdef73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33721 } }
I20260925 13:17:44.029330 5057 raft_consensus.cc:374] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:44.029351 5057 raft_consensus.cc:729] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b74819e38db24656aaa2ed0a1afdef73, State: Initialized, Role: FOLLOWER
I20260925 13:17:44.029474 5057 consensus_queue.cc:260] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74819e38db24656aaa2ed0a1afdef73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33721 } }
I20260925 13:17:44.029592 5057 raft_consensus.cc:388] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:44.029629 5057 raft_consensus.cc:482] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:44.029675 5057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:44.030457 5057 raft_consensus.cc:504] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74819e38db24656aaa2ed0a1afdef73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33721 } }
I20260925 13:17:44.030542 5057 leader_election.cc:302] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b74819e38db24656aaa2ed0a1afdef73; no voters:
I20260925 13:17:44.030692 5057 leader_election.cc:288] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:44.030781 5062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:44.030933 5057 sys_catalog.cc:564] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:44.030937 5062 raft_consensus.cc:686] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [term 1 LEADER]: Becoming Leader. State: Replica: b74819e38db24656aaa2ed0a1afdef73, State: Running, Role: LEADER
I20260925 13:17:44.031113 5062 consensus_queue.cc:237] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74819e38db24656aaa2ed0a1afdef73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33721 } }
I20260925 13:17:44.031508 5064 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b74819e38db24656aaa2ed0a1afdef73. Latest consensus state: current_term: 1 leader_uuid: "b74819e38db24656aaa2ed0a1afdef73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74819e38db24656aaa2ed0a1afdef73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33721 } } }
I20260925 13:17:44.031679 5064 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:44.031502 5063 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b74819e38db24656aaa2ed0a1afdef73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74819e38db24656aaa2ed0a1afdef73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33721 } } }
I20260925 13:17:44.031894 5071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:44.032083 5063 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:44.032500 5071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:44.033886 5071 catalog_manager.cc:1347] Generated new cluster ID: 81f1179fe65449c18dd01a16342e72d6
I20260925 13:17:44.033937 5071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:17:44.047724 5071 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:44.047926 5071 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:44.055642 5071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73: Generated new TSK 0
I20260925 13:17:44.055864 5071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:17:44.061065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33721
--builtin_ntp_servers=127.0.98.20:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 13:17:44.140697 5081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:44.140909 5081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:44.140939 5081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:44.142123 5081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:44.142164 5081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:44.142200 5081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:17:44.143569 5081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:44.143795 5081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:44.144061 5081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:44.148783 5086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:44.148859 5087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:44.148825 5089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:44.148993 5081 server_base.cc:1034] running on GCE node
I20260925 13:17:44.149194 5081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:44.149446 5081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:44.150599 5081 hybrid_clock.cc:648] HybridClock initialized: now 1790342264150591 us; error 38 us; skew 500 ppm
I20260925 13:17:44.151839 5081 webserver.cc:460] Webserver started at http://127.0.98.1:39025/ using document root <none> and password file <none>
I20260925 13:17:44.152076 5081 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:44.152122 5081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:44.152249 5081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:44.153348 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "adbd9f614f094ef49599a14bb121fba0"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "c2b373fb5c48f94f2bb6ce8e8a11d04a"
server_key_iv: "b8e8e746ff3357c4cbdc71e572732bfb"
server_key_version: "encryptionkey@0"
I20260925 13:17:44.153726 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "adbd9f614f094ef49599a14bb121fba0"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "c2b373fb5c48f94f2bb6ce8e8a11d04a"
server_key_iv: "b8e8e746ff3357c4cbdc71e572732bfb"
server_key_version: "encryptionkey@0"
I20260925 13:17:44.154985 5081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:17:44.155683 5095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:44.155841 5081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:17:44.155910 5081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "adbd9f614f094ef49599a14bb121fba0"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "c2b373fb5c48f94f2bb6ce8e8a11d04a"
server_key_iv: "b8e8e746ff3357c4cbdc71e572732bfb"
server_key_version: "encryptionkey@0"
I20260925 13:17:44.155979 5081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:44.159278 5081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:44.159531 5081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:44.159646 5081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:44.159850 5081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:44.160135 5081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:44.160164 5081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:44.160192 5081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:44.160214 5081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:44.165835 5081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43983
I20260925 13:17:44.165882 5208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43983 every 8 connection(s)
I20260925 13:17:44.166200 5081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:17:44.168903 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.166844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41727 (local address 127.0.98.62:33721)
0925 13:17:44.166892 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:44.166894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:44.167122 (+ 228us) server_negotiation.cc:407] Connection header received
0925 13:17:44.167589 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:44.167590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:44.167597 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:44.167662 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:44.168718 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:44.168720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:44.168720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:44.168724 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:44.168738 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:44.168742 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:44.168785 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:44.168840 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 13:17:44.168851 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:17:44.169078 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.166828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33721 (local address 127.0.98.1:41727)
0925 13:17:44.167002 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:44.167044 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:44.167499 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:44.167726 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:44.167729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:44.167737 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:44.168578 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:17:44.168578 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:44.168608 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:17:44.168637 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:44.168785 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:44.168788 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:44.168790 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:17:44.168842 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 13:17:44.168891 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260925 13:17:44.169571 5209 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33721
I20260925 13:17:44.169657 5209 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:44.169793 5209 heartbeater.cc:511] Master 127.0.98.62:33721 requested a full tablet report, sending...
I20260925 13:17:44.170262 5021 ts_manager.cc:194] Registered new tserver with Master: adbd9f614f094ef49599a14bb121fba0 (127.0.98.1:43983)
I20260925 13:17:44.175851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 5081
I20260925 13:17:44.175946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:17:44.176110 392 external_mini_cluster.cc:1428] Setting key e89959d17662d365019ce4a4a03bfa60
I20260925 13:17:44.176944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33721
--builtin_ntp_servers=127.0.98.20:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 13:17:44.257215 5212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:44.257450 5212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:44.257483 5212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:44.258808 5212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:44.258853 5212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:44.258889 5212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:17:44.260295 5212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:44.260547 5212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:44.260824 5212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:44.266539 5218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:44.266539 5217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:44.266547 5220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:44.266709 5212 server_base.cc:1034] running on GCE node
I20260925 13:17:44.266956 5212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:44.267164 5212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:44.268321 5212 hybrid_clock.cc:648] HybridClock initialized: now 1790342264268309 us; error 29 us; skew 500 ppm
I20260925 13:17:44.269657 5212 webserver.cc:460] Webserver started at http://127.0.98.2:42829/ using document root <none> and password file <none>
I20260925 13:17:44.269887 5212 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:44.269934 5212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:44.270056 5212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:44.271113 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "e5fcd1cdb7784c7c81015658f578e64d"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "1fb7a96f4d19e82c18b6d7141469564e"
server_key_iv: "abed7fe58678a0389adc0822680bd6d1"
server_key_version: "encryptionkey@0"
I20260925 13:17:44.271432 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e5fcd1cdb7784c7c81015658f578e64d"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "1fb7a96f4d19e82c18b6d7141469564e"
server_key_iv: "abed7fe58678a0389adc0822680bd6d1"
server_key_version: "encryptionkey@0"
I20260925 13:17:44.272646 5212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:44.273365 5226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:44.273510 5212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:44.273619 5212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "e5fcd1cdb7784c7c81015658f578e64d"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "1fb7a96f4d19e82c18b6d7141469564e"
server_key_iv: "abed7fe58678a0389adc0822680bd6d1"
server_key_version: "encryptionkey@0"
I20260925 13:17:44.273685 5212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:44.276926 5212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:44.277191 5212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:44.277313 5212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:44.277503 5212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:44.277827 5212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:44.277864 5212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:44.277885 5212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:44.277907 5212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:44.283345 5212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46625
I20260925 13:17:44.283396 5339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46625 every 8 connection(s)
I20260925 13:17:44.283718 5212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:17:44.286687 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.284443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59279 (local address 127.0.98.62:33721)
0925 13:17:44.284507 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:44.284508 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:44.284715 (+ 207us) server_negotiation.cc:407] Connection header received
0925 13:17:44.285206 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:44.285207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:44.285215 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:44.285268 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:44.286448 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:44.286449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:44.286450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:44.286455 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:44.286470 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:44.286474 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:44.286523 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:44.286599 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 13:17:44.286614 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 13:17:44.286887 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.284417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33721 (local address 127.0.98.2:59279)
0925 13:17:44.284599 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:44.284634 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:44.285110 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:44.285350 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:44.285354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:44.285362 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:44.286281 (+ 919us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:17:44.286282 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:44.286326 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:17:44.286358 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:44.286528 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:44.286532 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:44.286536 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:44.286605 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 13:17:44.286670 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260925 13:17:44.287351 5340 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33721
I20260925 13:17:44.287429 5340 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:44.287569 5340 heartbeater.cc:511] Master 127.0.98.62:33721 requested a full tablet report, sending...
I20260925 13:17:44.287926 5021 ts_manager.cc:194] Registered new tserver with Master: e5fcd1cdb7784c7c81015658f578e64d (127.0.98.2:46625)
I20260925 13:17:44.291357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 5212
I20260925 13:17:44.291437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:17:44.291579 392 external_mini_cluster.cc:1428] Setting key 359d83456733c206329cfd3e3e437c64
I20260925 13:17:44.292474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33721
--builtin_ntp_servers=127.0.98.20:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 13:17:44.373625 5343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:44.373847 5343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:44.373937 5343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:44.375209 5343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:44.375252 5343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:44.375288 5343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:17:44.376724 5343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:44.376976 5343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:44.377274 5343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:44.382114 5348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:44.382112 5349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:44.382114 5351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:44.382131 5343 server_base.cc:1034] running on GCE node
I20260925 13:17:44.382540 5343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:44.382753 5343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:44.383912 5343 hybrid_clock.cc:648] HybridClock initialized: now 1790342264383893 us; error 32 us; skew 500 ppm
I20260925 13:17:44.385223 5343 webserver.cc:460] Webserver started at http://127.0.98.3:46321/ using document root <none> and password file <none>
I20260925 13:17:44.385457 5343 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:44.385506 5343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:44.385660 5343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:44.386811 5343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "fa6e2045ffce4cdabae713a997af7450"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "68d93c249a129c16c3a84a4b007f10fb"
server_key_iv: "214be2d200d17614dd7296262ea46c58"
server_key_version: "encryptionkey@0"
I20260925 13:17:44.387143 5343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fa6e2045ffce4cdabae713a997af7450"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "68d93c249a129c16c3a84a4b007f10fb"
server_key_iv: "214be2d200d17614dd7296262ea46c58"
server_key_version: "encryptionkey@0"
I20260925 13:17:44.388494 5343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:44.389384 5357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:44.389575 5343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:44.389670 5343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "fa6e2045ffce4cdabae713a997af7450"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "68d93c249a129c16c3a84a4b007f10fb"
server_key_iv: "214be2d200d17614dd7296262ea46c58"
server_key_version: "encryptionkey@0"
I20260925 13:17:44.389743 5343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:44.393095 5343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:44.393361 5343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:44.393481 5343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:44.393692 5343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:44.393989 5343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:44.394021 5343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:44.394047 5343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:44.394070 5343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:44.399374 5343 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33959
I20260925 13:17:44.399428 5470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33959 every 8 connection(s)
I20260925 13:17:44.399742 5343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:17:44.402712 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.400498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45957 (local address 127.0.98.62:33721)
0925 13:17:44.400551 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:44.400553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:44.400749 (+ 196us) server_negotiation.cc:407] Connection header received
0925 13:17:44.401234 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:44.401235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:44.401241 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:44.401307 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:44.402460 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:44.402462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:44.402462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:44.402466 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:44.402481 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:44.402486 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:44.402536 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:44.402630 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 13:17:44.402645 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 13:17:44.402851 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.400447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33721 (local address 127.0.98.3:45957)
0925 13:17:44.400625 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:44.400666 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:44.401141 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:44.401377 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:44.401381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:44.401389 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:44.402298 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:17:44.402299 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:44.402329 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:17:44.402361 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:44.402537 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:44.402539 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:44.402542 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:44.402600 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 13:17:44.402652 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260925 13:17:44.403379 5471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33721
I20260925 13:17:44.403476 5471 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:44.403647 5471 heartbeater.cc:511] Master 127.0.98.62:33721 requested a full tablet report, sending...
I20260925 13:17:44.404042 5021 ts_manager.cc:194] Registered new tserver with Master: fa6e2045ffce4cdabae713a997af7450 (127.0.98.3:33959)
I20260925 13:17:44.407034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 5343
I20260925 13:17:44.407141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:17:44.407327 392 external_mini_cluster.cc:1428] Setting key 42f3160eb038b63ce98260612a553ad1
I20260925 13:17:44.408109 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:17:44.410550 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.409465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.0.98.62:33721)
0925 13:17:44.409579 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:44.409581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:44.409623 (+ 42us) server_negotiation.cc:407] Connection header received
0925 13:17:44.409694 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:44.409695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:44.409700 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:44.409749 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:44.410384 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:44.410385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:44.410385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:44.410387 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:44.410396 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:44.410399 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:44.410430 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:44.410468 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 13:17:44.410478 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 13:17:44.412135 5021 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:17:44.413695 5021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:44.417768 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.415787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46625 (local address 127.0.0.1:51136)
0925 13:17:44.415938 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:44.416012 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:44.416098 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:44.416375 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:44.416380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:44.416388 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:44.417236 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:17:44.417236 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:44.417254 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:17:44.417280 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:44.417576 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:44.417577 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:44.417578 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 13:17:44.417646 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 13:17:44.417660 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260925 13:17:44.417881 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.416048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.0.98.2:46625)
0925 13:17:44.416223 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:44.416229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:44.416239 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:17:44.416266 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:44.416268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:44.416278 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:44.416348 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:44.417373 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:44.417384 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:44.417385 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:44.417390 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:44.417428 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:44.417472 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:44.417528 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:44.417718 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 13:17:44.417734 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260925 13:17:44.417951 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.416048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60260 (local address 127.0.98.3:33959)
0925 13:17:44.416223 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:44.416229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:44.416763 (+ 534us) server_negotiation.cc:407] Connection header received
0925 13:17:44.416799 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:44.416802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:44.416814 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:44.416897 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:44.417372 (+ 475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:44.417382 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:44.417383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:44.417387 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:44.417431 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:44.417462 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:44.417512 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:44.417844 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 13:17:44.417859 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260925 13:17:44.417971 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.415787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33959 (local address 127.0.0.1:60260)
0925 13:17:44.416678 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:44.416688 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:44.416740 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:44.416906 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:44.416907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:44.416909 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:44.417267 (+ 358us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:17:44.417267 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:44.417274 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:17:44.417287 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:44.417560 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:44.417567 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:44.417570 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:44.417694 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 13:17:44.417708 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260925 13:17:44.418680 5405 tablet_service.cc:1467] Processing CreateTablet for tablet 0b5dfac3d7634d0f82c68292088a24b4 (DEFAULT_TABLE table=test-table [id=a656da3832674ca1b647f1997f6f4f74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:44.418814 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 0b5dfac3d7634d0f82c68292088a24b4 (DEFAULT_TABLE table=test-table [id=a656da3832674ca1b647f1997f6f4f74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:44.419055 5405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b5dfac3d7634d0f82c68292088a24b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:44.419138 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b5dfac3d7634d0f82c68292088a24b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:44.422674 5484 tablet_bootstrap.cc:492] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d: Bootstrap starting.
I20260925 13:17:44.422884 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.419882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43983 (local address 127.0.0.1:48502)
0925 13:17:44.420424 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:44.420436 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:44.420477 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:44.421522 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:44.421523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:44.421526 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:44.422514 (+ 988us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:17:44.422514 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:44.422522 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:17:44.422539 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:44.422781 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:44.422782 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:44.422783 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 13:17:44.422808 (+ 25us) client_negotiation.cc:240] Negotiation successful
0925 13:17:44.422823 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523}
I20260925 13:17:44.423116 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.420959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.0.98.1:43983)
0925 13:17:44.421366 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:44.421371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:44.421381 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:17:44.421407 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:44.421409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:44.421419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:44.421703 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:44.422634 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:44.422641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:44.422641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:44.422646 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:44.422679 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:44.422707 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:44.422997 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:44.423024 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 13:17:44.423040 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":178,"threads_started":1}
I20260925 13:17:44.423540 5484 tablet_bootstrap.cc:654] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:44.423601 5485 tablet_bootstrap.cc:492] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450: Bootstrap starting.
I20260925 13:17:44.423826 5484 log.cc:824] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:44.423856 5143 tablet_service.cc:1467] Processing CreateTablet for tablet 0b5dfac3d7634d0f82c68292088a24b4 (DEFAULT_TABLE table=test-table [id=a656da3832674ca1b647f1997f6f4f74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:44.424203 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b5dfac3d7634d0f82c68292088a24b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:44.424237 5485 tablet_bootstrap.cc:654] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:44.424535 5484 tablet_bootstrap.cc:492] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d: No bootstrap required, opened a new log
I20260925 13:17:44.424521 5485 log.cc:824] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:44.424593 5484 ts_tablet_manager.cc:1397] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:17:44.425233 5485 tablet_bootstrap.cc:492] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450: No bootstrap required, opened a new log
I20260925 13:17:44.425314 5485 ts_tablet_manager.cc:1397] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:17:44.426575 5484 raft_consensus.cc:348] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6e2045ffce4cdabae713a997af7450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33959 } } peers { permanent_uuid: "e5fcd1cdb7784c7c81015658f578e64d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46625 } } peers { permanent_uuid: "adbd9f614f094ef49599a14bb121fba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43983 } }
I20260925 13:17:44.426709 5484 raft_consensus.cc:374] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:44.426740 5484 raft_consensus.cc:729] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5fcd1cdb7784c7c81015658f578e64d, State: Initialized, Role: FOLLOWER
I20260925 13:17:44.426873 5484 consensus_queue.cc:260] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6e2045ffce4cdabae713a997af7450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33959 } } peers { permanent_uuid: "e5fcd1cdb7784c7c81015658f578e64d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46625 } } peers { permanent_uuid: "adbd9f614f094ef49599a14bb121fba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43983 } }
I20260925 13:17:44.427145 5340 heartbeater.cc:503] Master 127.0.98.62:33721 was elected leader, sending a full tablet report...
I20260925 13:17:44.427145 5484 ts_tablet_manager.cc:1428] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:17:44.427655 5489 tablet_bootstrap.cc:492] T 0b5dfac3d7634d0f82c68292088a24b4 P adbd9f614f094ef49599a14bb121fba0: Bootstrap starting.
I20260925 13:17:44.428267 5489 tablet_bootstrap.cc:654] T 0b5dfac3d7634d0f82c68292088a24b4 P adbd9f614f094ef49599a14bb121fba0: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:44.428161 5485 raft_consensus.cc:348] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6e2045ffce4cdabae713a997af7450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33959 } } peers { permanent_uuid: "e5fcd1cdb7784c7c81015658f578e64d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46625 } } peers { permanent_uuid: "adbd9f614f094ef49599a14bb121fba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43983 } }
I20260925 13:17:44.428325 5485 raft_consensus.cc:374] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:44.428367 5485 raft_consensus.cc:729] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6e2045ffce4cdabae713a997af7450, State: Initialized, Role: FOLLOWER
I20260925 13:17:44.428486 5485 consensus_queue.cc:260] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6e2045ffce4cdabae713a997af7450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33959 } } peers { permanent_uuid: "e5fcd1cdb7784c7c81015658f578e64d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46625 } } peers { permanent_uuid: "adbd9f614f094ef49599a14bb121fba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43983 } }
I20260925 13:17:44.428579 5489 log.cc:824] T 0b5dfac3d7634d0f82c68292088a24b4 P adbd9f614f094ef49599a14bb121fba0: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:44.428728 5485 ts_tablet_manager.cc:1428] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:17:44.428802 5471 heartbeater.cc:503] Master 127.0.98.62:33721 was elected leader, sending a full tablet report...
I20260925 13:17:44.429260 5489 tablet_bootstrap.cc:492] T 0b5dfac3d7634d0f82c68292088a24b4 P adbd9f614f094ef49599a14bb121fba0: No bootstrap required, opened a new log
I20260925 13:17:44.429328 5489 ts_tablet_manager.cc:1397] T 0b5dfac3d7634d0f82c68292088a24b4 P adbd9f614f094ef49599a14bb121fba0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:17:44.431348 5489 raft_consensus.cc:348] T 0b5dfac3d7634d0f82c68292088a24b4 P adbd9f614f094ef49599a14bb121fba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6e2045ffce4cdabae713a997af7450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33959 } } peers { permanent_uuid: "e5fcd1cdb7784c7c81015658f578e64d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46625 } } peers { permanent_uuid: "adbd9f614f094ef49599a14bb121fba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43983 } }
I20260925 13:17:44.431504 5489 raft_consensus.cc:374] T 0b5dfac3d7634d0f82c68292088a24b4 P adbd9f614f094ef49599a14bb121fba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:44.431527 5489 raft_consensus.cc:729] T 0b5dfac3d7634d0f82c68292088a24b4 P adbd9f614f094ef49599a14bb121fba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbd9f614f094ef49599a14bb121fba0, State: Initialized, Role: FOLLOWER
I20260925 13:17:44.431610 5489 consensus_queue.cc:260] T 0b5dfac3d7634d0f82c68292088a24b4 P adbd9f614f094ef49599a14bb121fba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6e2045ffce4cdabae713a997af7450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33959 } } peers { permanent_uuid: "e5fcd1cdb7784c7c81015658f578e64d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46625 } } peers { permanent_uuid: "adbd9f614f094ef49599a14bb121fba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43983 } }
I20260925 13:17:44.431803 5489 ts_tablet_manager.cc:1428] T 0b5dfac3d7634d0f82c68292088a24b4 P adbd9f614f094ef49599a14bb121fba0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 13:17:44.431860 5209 heartbeater.cc:503] Master 127.0.98.62:33721 was elected leader, sending a full tablet report...
W20260925 13:17:44.534214 5341 tablet.cc:2369] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:17:44.607321 5488 raft_consensus.cc:482] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:44.607455 5488 raft_consensus.cc:504] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6e2045ffce4cdabae713a997af7450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33959 } } peers { permanent_uuid: "e5fcd1cdb7784c7c81015658f578e64d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46625 } } peers { permanent_uuid: "adbd9f614f094ef49599a14bb121fba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43983 } }
I20260925 13:17:44.607779 5488 leader_election.cc:288] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6e2045ffce4cdabae713a997af7450 (127.0.98.3:33959), adbd9f614f094ef49599a14bb121fba0 (127.0.98.1:43983)
I20260925 13:17:44.609118 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.607984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33959 (local address 127.0.98.2:37377)
0925 13:17:44.608089 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:44.608102 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:44.608152 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:44.608306 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:44.608308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:44.608311 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:44.608783 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:17:44.608784 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:44.608791 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:17:44.608804 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:44.608995 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:44.608996 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:44.608996 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 13:17:44.609039 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:17:44.609051 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 13:17:44.609136 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.608084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37377 (local address 127.0.98.3:33959)
0925 13:17:44.608134 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:44.608136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:44.608145 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:44.608224 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:44.608225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:44.608231 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:44.608287 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:44.608905 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:44.608907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:44.608907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:44.608912 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:44.608930 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:44.608934 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:44.608984 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:44.609051 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 13:17:44.609065 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 13:17:44.609225 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.608089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39421 (local address 127.0.98.1:43983)
0925 13:17:44.608130 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:44.608132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:44.608313 (+ 181us) server_negotiation.cc:407] Connection header received
0925 13:17:44.608347 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:44.608348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:44.608355 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:44.608417 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:44.609020 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:44.609022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:44.609023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:44.609027 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:44.609044 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:44.609048 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:44.609100 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:44.609151 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 13:17:44.609163 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:17:44.609387 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.608008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43983 (local address 127.0.98.2:39421)
0925 13:17:44.608202 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:44.608212 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:44.608274 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:44.608433 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:44.608434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:44.608436 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:44.608930 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:17:44.608930 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:44.608935 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:17:44.608944 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:44.609107 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:44.609107 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:44.609108 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 13:17:44.609151 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:17:44.609161 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":112,"threads_started":1}
I20260925 13:17:44.609627 5425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5dfac3d7634d0f82c68292088a24b4" candidate_uuid: "e5fcd1cdb7784c7c81015658f578e64d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6e2045ffce4cdabae713a997af7450" is_pre_election: true
I20260925 13:17:44.609790 5425 raft_consensus.cc:2454] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5fcd1cdb7784c7c81015658f578e64d in term 0.
I20260925 13:17:44.609755 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5dfac3d7634d0f82c68292088a24b4" candidate_uuid: "e5fcd1cdb7784c7c81015658f578e64d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbd9f614f094ef49599a14bb121fba0" is_pre_election: true
I20260925 13:17:44.609898 5163 raft_consensus.cc:2454] T 0b5dfac3d7634d0f82c68292088a24b4 P adbd9f614f094ef49599a14bb121fba0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5fcd1cdb7784c7c81015658f578e64d in term 0.
I20260925 13:17:44.610030 5230 leader_election.cc:302] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e5fcd1cdb7784c7c81015658f578e64d, fa6e2045ffce4cdabae713a997af7450; no voters:
I20260925 13:17:44.610165 5488 raft_consensus.cc:2781] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:44.610219 5488 raft_consensus.cc:482] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:44.610241 5488 raft_consensus.cc:3037] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:44.610981 5488 raft_consensus.cc:504] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6e2045ffce4cdabae713a997af7450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33959 } } peers { permanent_uuid: "e5fcd1cdb7784c7c81015658f578e64d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46625 } } peers { permanent_uuid: "adbd9f614f094ef49599a14bb121fba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43983 } }
I20260925 13:17:44.611111 5488 leader_election.cc:288] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [CANDIDATE]: Term 1 election: Requested vote from peers fa6e2045ffce4cdabae713a997af7450 (127.0.98.3:33959), adbd9f614f094ef49599a14bb121fba0 (127.0.98.1:43983)
I20260925 13:17:44.611275 5425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5dfac3d7634d0f82c68292088a24b4" candidate_uuid: "e5fcd1cdb7784c7c81015658f578e64d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6e2045ffce4cdabae713a997af7450"
I20260925 13:17:44.611343 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5dfac3d7634d0f82c68292088a24b4" candidate_uuid: "e5fcd1cdb7784c7c81015658f578e64d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbd9f614f094ef49599a14bb121fba0"
I20260925 13:17:44.611353 5425 raft_consensus.cc:3037] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:44.611438 5163 raft_consensus.cc:3037] T 0b5dfac3d7634d0f82c68292088a24b4 P adbd9f614f094ef49599a14bb121fba0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:44.612051 5425 raft_consensus.cc:2454] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5fcd1cdb7784c7c81015658f578e64d in term 1.
I20260925 13:17:44.612167 5163 raft_consensus.cc:2454] T 0b5dfac3d7634d0f82c68292088a24b4 P adbd9f614f094ef49599a14bb121fba0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5fcd1cdb7784c7c81015658f578e64d in term 1.
I20260925 13:17:44.612202 5230 leader_election.cc:302] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e5fcd1cdb7784c7c81015658f578e64d, fa6e2045ffce4cdabae713a997af7450; no voters:
I20260925 13:17:44.612339 5488 raft_consensus.cc:2781] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:44.612509 5488 raft_consensus.cc:686] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [term 1 LEADER]: Becoming Leader. State: Replica: e5fcd1cdb7784c7c81015658f578e64d, State: Running, Role: LEADER
I20260925 13:17:44.612613 5488 consensus_queue.cc:237] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6e2045ffce4cdabae713a997af7450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33959 } } peers { permanent_uuid: "e5fcd1cdb7784c7c81015658f578e64d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46625 } } peers { permanent_uuid: "adbd9f614f094ef49599a14bb121fba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43983 } }
I20260925 13:17:44.613466 5020 catalog_manager.cc:5495] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d reported cstate change: term changed from 0 to 1, leader changed from <none> to e5fcd1cdb7784c7c81015658f578e64d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e5fcd1cdb7784c7c81015658f578e64d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6e2045ffce4cdabae713a997af7450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5fcd1cdb7784c7c81015658f578e64d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adbd9f614f094ef49599a14bb121fba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43983 } health_report { overall_health: UNKNOWN } } }
I20260925 13:17:44.647186 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.646301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51146 (local address 127.0.98.2:46625)
0925 13:17:44.646362 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:44.646364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:44.646373 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:44.646431 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:44.646432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:44.646444 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:44.646489 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:44.647001 (+ 512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:44.647002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:44.647002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:44.647006 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:44.647020 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:44.647024 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:44.647066 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:44.647099 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 13:17:44.647110 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 13:17:44.649029 5425 raft_consensus.cc:1261] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450 [term 1 FOLLOWER]: Refusing update from remote peer e5fcd1cdb7784c7c81015658f578e64d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:44.649040 5163 raft_consensus.cc:1261] T 0b5dfac3d7634d0f82c68292088a24b4 P adbd9f614f094ef49599a14bb121fba0 [term 1 FOLLOWER]: Refusing update from remote peer e5fcd1cdb7784c7c81015658f578e64d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:44.649366 5488 consensus_queue.cc:1035] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6e2045ffce4cdabae713a997af7450" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:44.649514 5494 consensus_queue.cc:1035] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [LEADER]: Connected to new peer: Peer: permanent_uuid: "adbd9f614f094ef49599a14bb121fba0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 13:17:44.650965 5472 tablet.cc:2369] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:17:44.652072 5497 mvcc.cc:205] Tried to move back new op lower bound from 7333241915999645696 to 7333241915853897728. Current Snapshot: MvccSnapshot[applied={T|T < 7333241915999645696}]
I20260925 13:17:44.652570 5499 mvcc.cc:205] Tried to move back new op lower bound from 7333241915999645696 to 7333241915853897728. Current Snapshot: MvccSnapshot[applied={T|T < 7333241915999645696}]
I20260925 13:17:44.652789 5498 mvcc.cc:205] Tried to move back new op lower bound from 7333241915999645696 to 7333241915853897728. Current Snapshot: MvccSnapshot[applied={T|T < 7333241915999645696}]
I20260925 13:17:44.654253 5019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:44.654345 5019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:44.655072 5019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b74819e38db24656aaa2ed0a1afdef73: Sending DeleteTablet for 3 replicas of tablet 0b5dfac3d7634d0f82c68292088a24b4
I20260925 13:17:44.655365 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b5dfac3d7634d0f82c68292088a24b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:44 UTC) from {username='slave'} at 127.0.0.1:51136
I20260925 13:17:44.655388 5405 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b5dfac3d7634d0f82c68292088a24b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:44 UTC) from {username='slave'} at 127.0.0.1:60260
I20260925 13:17:44.655457 5143 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b5dfac3d7634d0f82c68292088a24b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:44 UTC) from {username='slave'} at 127.0.0.1:48502
I20260925 13:17:44.655678 5507 tablet_replica.cc:330] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d: stopping tablet replica
I20260925 13:17:44.655727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 5081
I20260925 13:17:44.655781 5507 raft_consensus.cc:2229] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:17:44.655912 5507 raft_consensus.cc:2258] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:44.656774 5507 ts_tablet_manager.cc:1905] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:44.658588 5507 ts_tablet_manager.cc:1918] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:44.658661 5507 log.cc:1196] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/0b5dfac3d7634d0f82c68292088a24b4
I20260925 13:17:44.658934 5507 ts_tablet_manager.cc:1939] T 0b5dfac3d7634d0f82c68292088a24b4 P e5fcd1cdb7784c7c81015658f578e64d: Deleting consensus metadata
I20260925 13:17:44.659324 5005 catalog_manager.cc:4841] TS e5fcd1cdb7784c7c81015658f578e64d (127.0.98.2:46625): tablet 0b5dfac3d7634d0f82c68292088a24b4 (table test-table [id=a656da3832674ca1b647f1997f6f4f74]) successfully deleted
W20260925 13:17:44.661782 5007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43983 (error 108)
I20260925 13:17:44.662256 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.662128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43983 (local address 127.0.0.1:48516)
0925 13:17:44.662178 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:44.662219 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 13:17:44.662285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 5212
I20260925 13:17:44.662312 5508 tablet_replica.cc:330] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450: stopping tablet replica
I20260925 13:17:44.662427 5508 raft_consensus.cc:2229] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:44.662539 5508 raft_consensus.cc:2258] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:44.663586 5508 ts_tablet_manager.cc:1905] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 13:17:44.664901 5007 catalog_manager.cc:4584] TS adbd9f614f094ef49599a14bb121fba0 (127.0.98.1:43983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b5dfac3d7634d0f82c68292088a24b4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43983: connect: Connection refused (error 111)
I20260925 13:17:44.665103 5508 ts_tablet_manager.cc:1918] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:44.665174 5508 log.cc:1196] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/0b5dfac3d7634d0f82c68292088a24b4
I20260925 13:17:44.665409 5508 ts_tablet_manager.cc:1939] T 0b5dfac3d7634d0f82c68292088a24b4 P fa6e2045ffce4cdabae713a997af7450: Deleting consensus metadata
I20260925 13:17:44.665930 5008 catalog_manager.cc:4841] TS fa6e2045ffce4cdabae713a997af7450 (127.0.98.3:33959): tablet 0b5dfac3d7634d0f82c68292088a24b4 (table test-table [id=a656da3832674ca1b647f1997f6f4f74]) successfully deleted
I20260925 13:17:44.668048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 5343
I20260925 13:17:44.672431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 4990
2026-09-25T13:17:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (776 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 13:17:44.690933 392 mini_oidc.cc:152] Starting JWKS server
I20260925 13:17:44.691970 392 webserver.cc:460] Webserver started at https://127.0.0.1:46421/ using document root <none> and password file <none>
I20260925 13:17:44.692049 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 13:17:44.692207 392 webserver.cc:460] Webserver started at http://127.0.0.1:34739/ using document root <none> and password file <none>
2026-09-25T13:17:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:44Z Disabled control of system clock
I20260925 13:17:44.699546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36973
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36973
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46421/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/testchainca.cert with env {}
W20260925 13:17:44.779771 5516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:44.780007 5516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:44.780031 5516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:44.781214 5516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:44.781250 5516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:44.781260 5516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:44.781271 5516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:44.781281 5516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 13:17:44.781292 5516 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 13:17:44.781302 5516 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46421/jwks.json
I20260925 13:17:44.782876 5516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46421/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:44.783090 5516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:44.783298 5516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:44.788012 5522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:44.788018 5524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:44.788050 5521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:44.788053 5516 server_base.cc:1034] running on GCE node
I20260925 13:17:44.788419 5516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:44.788697 5516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:44.789829 5516 hybrid_clock.cc:648] HybridClock initialized: now 1790342264789814 us; error 29 us; skew 500 ppm
I20260925 13:17:44.797003 5516 webserver.cc:460] Webserver started at http://127.0.98.62:35465/ using document root <none> and password file <none>
I20260925 13:17:44.797243 5516 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:44.797294 5516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:44.797426 5516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:44.798574 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "ea30f215e4234d0b9c69c7d4d3d028d2"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "929870e0b2dca3be65d2ed7a18ad2cc4"
server_key_iv: "0e847325c2b8a6602bfb472c74700d5f"
server_key_version: "encryptionkey@0"
I20260925 13:17:44.798915 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea30f215e4234d0b9c69c7d4d3d028d2"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "929870e0b2dca3be65d2ed7a18ad2cc4"
server_key_iv: "0e847325c2b8a6602bfb472c74700d5f"
server_key_version: "encryptionkey@0"
I20260925 13:17:44.800124 5516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 13:17:44.800791 5530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:44.800940 5516 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:17:44.801024 5516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "ea30f215e4234d0b9c69c7d4d3d028d2"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "929870e0b2dca3be65d2ed7a18ad2cc4"
server_key_iv: "0e847325c2b8a6602bfb472c74700d5f"
server_key_version: "encryptionkey@0"
I20260925 13:17:44.801090 5516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:44.814399 5516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:44.814702 5516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:44.814829 5516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:44.818476 5516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36973
I20260925 13:17:44.818511 5582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36973 every 8 connection(s)
I20260925 13:17:44.818833 5516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:44.819742 5583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:44.822057 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2: Bootstrap starting.
I20260925 13:17:44.822589 5583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:44.822815 5583 log.cc:824] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:44.823436 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2: No bootstrap required, opened a new log
I20260925 13:17:44.823887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 5516
I20260925 13:17:44.823982 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:44.824136 392 external_mini_cluster.cc:1428] Setting key b8b25aca98f689944ff8c750328706ee
I20260925 13:17:44.825466 5583 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea30f215e4234d0b9c69c7d4d3d028d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36973 } }
I20260925 13:17:44.825675 5583 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:44.825719 5583 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea30f215e4234d0b9c69c7d4d3d028d2, State: Initialized, Role: FOLLOWER
I20260925 13:17:44.825840 5583 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea30f215e4234d0b9c69c7d4d3d028d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36973 } }
I20260925 13:17:44.825925 5583 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:44.825947 5583 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:44.825994 5583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:44.826525 5583 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea30f215e4234d0b9c69c7d4d3d028d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36973 } }
I20260925 13:17:44.826609 5583 leader_election.cc:302] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea30f215e4234d0b9c69c7d4d3d028d2; no voters:
I20260925 13:17:44.826758 5583 leader_election.cc:288] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:44.826800 5588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:44.826957 5588 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [term 1 LEADER]: Becoming Leader. State: Replica: ea30f215e4234d0b9c69c7d4d3d028d2, State: Running, Role: LEADER
I20260925 13:17:44.827019 5583 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:44.827042 5588 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea30f215e4234d0b9c69c7d4d3d028d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36973 } }
I20260925 13:17:44.827379 5588 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea30f215e4234d0b9c69c7d4d3d028d2. Latest consensus state: current_term: 1 leader_uuid: "ea30f215e4234d0b9c69c7d4d3d028d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea30f215e4234d0b9c69c7d4d3d028d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36973 } } }
I20260925 13:17:44.827464 5588 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:44.827394 5590 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea30f215e4234d0b9c69c7d4d3d028d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea30f215e4234d0b9c69c7d4d3d028d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36973 } } }
I20260925 13:17:44.827708 5595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:44.827735 5590 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:44.828250 5595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:44.830336 5595 catalog_manager.cc:1347] Generated new cluster ID: 47c7c7baafce4c528ec48f5832e2d05a
I20260925 13:17:44.830390 5595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:17:44.830809 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.824626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43622 (local address 127.0.98.62:36973)
0925 13:17:44.824851 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:44.824862 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:44.824898 (+ 36us) server_negotiation.cc:407] Connection header received
0925 13:17:44.825431 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:44.825441 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:44.825527 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:44.825705 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:44.825891 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:44.827375 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:44.828522 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:44.829065 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:44.829176 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:44.829928 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:44.829932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:44.829933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:44.830135 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:44.830177 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:44.830192 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:44.830249 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:44.830532 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 13:17:44.830609 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260925 13:17:44.831699 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:17:44.840553 5595 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:44.841074 5595 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:44.844810 5595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea30f215e4234d0b9c69c7d4d3d028d2: Generated new TSK 0
I20260925 13:17:44.844975 5595 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 13:17:44.866822 5613 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 13:17:44.866936 5613 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260925 13:17:44.867437 5612 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:44.852448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36973 (local address 127.0.0.1:43634)
0925 13:17:44.852619 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:44.852629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:44.852695 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:44.852871 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:44.852872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:44.852873 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 13:17:44.852936 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:44.852938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:44.853598 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:44.853599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:44.854218 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:44.854220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:44.854552 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:44.854557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:44.854620 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:44.854642 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 13:17:44.867103 (+ 12461us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 13:17:44.867184 (+ 81us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36973: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:36973: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 13:17:44.867723 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:44.852551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.0.98.62:36973)
0925 13:17:44.852604 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:44.852606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:44.852683 (+ 77us) server_negotiation.cc:407] Connection header received
0925 13:17:44.852788 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:44.852789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:44.852801 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:44.852864 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0925 13:17:44.852992 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:44.853486 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:44.854300 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:44.854497 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:44.854549 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:44.854752 (+ 203us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 13:17:44.866946 (+ 12194us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 13:17:44.867661 (+ 715us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43634: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 13:17:44.867745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 5516
W20260925 13:17:44.867772 5587 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43634: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
2026-09-25T13:17:44Z chronyd exiting
I20260925 13:17:44.903824 392 test_util.cc:175] -----------------------------------------------
I20260925 13:17:44.903914 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790342239834770-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 13:17:44.904659 392 mini_oidc.cc:152] Starting JWKS server
I20260925 13:17:44.905344 392 webserver.cc:460] Webserver started at https://127.0.0.1:41021/ using document root <none> and password file <none>
I20260925 13:17:44.905388 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 13:17:44.905524 392 webserver.cc:460] Webserver started at http://127.0.0.1:34931/ using document root <none> and password file <none>
2026-09-25T13:17:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:44Z Disabled control of system clock
I20260925 13:17:44.912590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33797
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33797
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41021/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/testchainca.cert with env {}
W20260925 13:17:44.992370 5619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:44.992594 5619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:44.992625 5619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:44.993856 5619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:44.993908 5619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:44.993925 5619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:44.993942 5619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:44.993956 5619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 13:17:44.993973 5619 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 13:17:44.993995 5619 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41021/jwks.json
I20260925 13:17:44.995556 5619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33797
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41021/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:44.995820 5619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:44.996030 5619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:45.000571 5624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:45.000564 5625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:45.000726 5619 server_base.cc:1034] running on GCE node
W20260925 13:17:45.000607 5627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:45.001077 5619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:45.001360 5619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:45.002539 5619 hybrid_clock.cc:648] HybridClock initialized: now 1790342265002511 us; error 37 us; skew 500 ppm
I20260925 13:17:45.003944 5619 webserver.cc:460] Webserver started at http://127.0.98.62:35895/ using document root <none> and password file <none>
I20260925 13:17:45.004173 5619 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:45.004226 5619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:45.004361 5619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:45.005430 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "32451d496e0b4c3187c897593ad63e80"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "6bf19b4ff8193b2524ea65028dcb7364"
server_key_iv: "19dea6a87c7ce682bb95c2d3caf1e3d6"
server_key_version: "encryptionkey@0"
I20260925 13:17:45.005837 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "32451d496e0b4c3187c897593ad63e80"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "6bf19b4ff8193b2524ea65028dcb7364"
server_key_iv: "19dea6a87c7ce682bb95c2d3caf1e3d6"
server_key_version: "encryptionkey@0"
I20260925 13:17:45.007251 5619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:45.008059 5633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:45.008225 5619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:45.008333 5619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "32451d496e0b4c3187c897593ad63e80"
format_stamp: "Formatted at 2026-09-25 13:17:44 on dist-test-slave-t8d9"
server_key: "6bf19b4ff8193b2524ea65028dcb7364"
server_key_iv: "19dea6a87c7ce682bb95c2d3caf1e3d6"
server_key_version: "encryptionkey@0"
I20260925 13:17:45.008409 5619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:45.026180 5619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:45.026559 5619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:45.026698 5619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:45.030349 5619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33797
I20260925 13:17:45.030383 5685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33797 every 8 connection(s)
I20260925 13:17:45.030704 5619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:45.031634 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:45.033891 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80: Bootstrap starting.
I20260925 13:17:45.034430 5686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:45.034648 5686 log.cc:824] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:45.035344 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80: No bootstrap required, opened a new log
I20260925 13:17:45.036695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 5619
I20260925 13:17:45.036790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:45.036981 392 external_mini_cluster.cc:1428] Setting key 41dbb165d233110f0ec04f28a7e1594e
I20260925 13:17:45.036945 5686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32451d496e0b4c3187c897593ad63e80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33797 } }
I20260925 13:17:45.037102 5686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:45.037146 5686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32451d496e0b4c3187c897593ad63e80, State: Initialized, Role: FOLLOWER
I20260925 13:17:45.037308 5686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32451d496e0b4c3187c897593ad63e80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33797 } }
I20260925 13:17:45.037400 5686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:45.037436 5686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:45.037483 5686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:45.038117 5686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32451d496e0b4c3187c897593ad63e80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33797 } }
I20260925 13:17:45.038229 5686 leader_election.cc:302] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32451d496e0b4c3187c897593ad63e80; no voters:
I20260925 13:17:45.038395 5686 leader_election.cc:288] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:45.038540 5691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:45.038651 5686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:45.038664 5691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [term 1 LEADER]: Becoming Leader. State: Replica: 32451d496e0b4c3187c897593ad63e80, State: Running, Role: LEADER
I20260925 13:17:45.038772 5691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32451d496e0b4c3187c897593ad63e80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33797 } }
I20260925 13:17:45.039117 5692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32451d496e0b4c3187c897593ad63e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32451d496e0b4c3187c897593ad63e80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33797 } } }
I20260925 13:17:45.039196 5692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:45.039345 5693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32451d496e0b4c3187c897593ad63e80. Latest consensus state: current_term: 1 leader_uuid: "32451d496e0b4c3187c897593ad63e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32451d496e0b4c3187c897593ad63e80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33797 } } }
I20260925 13:17:45.039404 5693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:45.039515 5700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:45.040151 5700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:45.042102 5700 catalog_manager.cc:1347] Generated new cluster ID: fe6ea45178df42f19e64ac6b0623c056
I20260925 13:17:45.042155 5700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:17:45.043066 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:45.037450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.0.98.62:33797)
0925 13:17:45.037708 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:45.037716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:45.037741 (+ 25us) server_negotiation.cc:407] Connection header received
0925 13:17:45.037849 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:45.037855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:45.037923 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:45.038037 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:45.038292 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.039702 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.040834 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.041420 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.041560 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:45.042362 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:45.042366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:45.042367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:45.042571 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:45.042612 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:45.042626 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:45.042685 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:45.042778 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 13:17:45.042855 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":150,"threads_started":1}
I20260925 13:17:45.043965 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:17:45.050994 5700 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:45.051541 5700 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:45.056686 5700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32451d496e0b4c3187c897593ad63e80: Generated new TSK 0
I20260925 13:17:45.056996 5700 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 13:17:45.080026 5716 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 13:17:45.080122 5716 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 13:17:45.080477 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:45.066358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40974 (local address 127.0.98.62:33797)
0925 13:17:45.066407 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:45.066409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:45.066521 (+ 112us) server_negotiation.cc:407] Connection header received
0925 13:17:45.066600 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:45.066601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:45.066614 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:45.066682 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0925 13:17:45.066801 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.067303 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.068064 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.068291 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.068346 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:45.068520 (+ 174us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 13:17:45.080243 (+ 11723us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 13:17:45.080407 (+ 164us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40974: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 13:17:45.080543 5690 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40974: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 13:17:45.080580 5715 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:45.066304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33797 (local address 127.0.0.1:40974)
0925 13:17:45.066460 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:45.066470 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:45.066532 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:45.066685 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:45.066686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:45.066687 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 13:17:45.066746 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:45.066748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.067383 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.067384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:45.067983 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:45.067985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.068351 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.068355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:45.068421 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:45.068426 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 13:17:45.080369 (+ 11943us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 13:17:45.080489 (+ 120us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33797: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260925 13:17:45.080868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 5619
2026-09-25T13:17:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (212 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 13:17:45.117741 392 mini_oidc.cc:152] Starting JWKS server
I20260925 13:17:45.118425 392 webserver.cc:460] Webserver started at https://127.0.0.1:40829/ using document root <none> and password file <none>
I20260925 13:17:45.118474 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 13:17:45.118614 392 webserver.cc:460] Webserver started at http://127.0.0.1:36383/ using document root <none> and password file <none>
2026-09-25T13:17:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:45Z Disabled control of system clock
I20260925 13:17:45.125802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46041
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46041
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40829/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 13:17:45.204888 5722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:45.205106 5722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:45.205127 5722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:45.206298 5722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:45.206336 5722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:45.206346 5722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:45.206357 5722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:45.206367 5722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 13:17:45.206378 5722 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 13:17:45.206388 5722 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40829/jwks.json
I20260925 13:17:45.207757 5722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40829/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:45.207974 5722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:45.208222 5722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:45.212839 5727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:45.212901 5722 server_base.cc:1034] running on GCE node
W20260925 13:17:45.212859 5730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:45.212839 5728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:45.213272 5722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:45.213647 5722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:45.214811 5722 hybrid_clock.cc:648] HybridClock initialized: now 1790342265214794 us; error 28 us; skew 500 ppm
I20260925 13:17:45.216110 5722 webserver.cc:460] Webserver started at http://127.0.98.62:35555/ using document root <none> and password file <none>
I20260925 13:17:45.216348 5722 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:45.216399 5722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:45.216529 5722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:45.217687 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "08dc25c7670b4b05822c2c02b87b9e88"
format_stamp: "Formatted at 2026-09-25 13:17:45 on dist-test-slave-t8d9"
server_key: "77b02157848e9f5af9200e13a4f08b98"
server_key_iv: "5cbfa50825ebda89b089a1ed6b2fdb4c"
server_key_version: "encryptionkey@0"
I20260925 13:17:45.218039 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "08dc25c7670b4b05822c2c02b87b9e88"
format_stamp: "Formatted at 2026-09-25 13:17:45 on dist-test-slave-t8d9"
server_key: "77b02157848e9f5af9200e13a4f08b98"
server_key_iv: "5cbfa50825ebda89b089a1ed6b2fdb4c"
server_key_version: "encryptionkey@0"
I20260925 13:17:45.219353 5722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:17:45.220108 5736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:45.220260 5722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:45.220348 5722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "08dc25c7670b4b05822c2c02b87b9e88"
format_stamp: "Formatted at 2026-09-25 13:17:45 on dist-test-slave-t8d9"
server_key: "77b02157848e9f5af9200e13a4f08b98"
server_key_iv: "5cbfa50825ebda89b089a1ed6b2fdb4c"
server_key_version: "encryptionkey@0"
I20260925 13:17:45.220409 5722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:45.227771 5722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:45.228076 5722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:45.228209 5722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:45.231994 5722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46041
I20260925 13:17:45.232024 5788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46041 every 8 connection(s)
I20260925 13:17:45.232371 5722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:45.233035 5789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:45.235762 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88: Bootstrap starting.
I20260925 13:17:45.236358 5789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:45.236619 5789 log.cc:824] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:45.237324 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88: No bootstrap required, opened a new log
I20260925 13:17:45.238992 5789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08dc25c7670b4b05822c2c02b87b9e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46041 } }
I20260925 13:17:45.239144 5789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:45.239166 5789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08dc25c7670b4b05822c2c02b87b9e88, State: Initialized, Role: FOLLOWER
I20260925 13:17:45.239296 5789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08dc25c7670b4b05822c2c02b87b9e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46041 } }
I20260925 13:17:45.239362 5789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:45.239382 5789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:45.239409 5789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:45.239872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 5722
I20260925 13:17:45.239969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:45.239977 5789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08dc25c7670b4b05822c2c02b87b9e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46041 } }
I20260925 13:17:45.240079 5789 leader_election.cc:302] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08dc25c7670b4b05822c2c02b87b9e88; no voters:
I20260925 13:17:45.240149 392 external_mini_cluster.cc:1428] Setting key 5d9a0b7daea4b570d30a24398edaa1b2
I20260925 13:17:45.240247 5789 leader_election.cc:288] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:45.240278 5792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:45.240393 5792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [term 1 LEADER]: Becoming Leader. State: Replica: 08dc25c7670b4b05822c2c02b87b9e88, State: Running, Role: LEADER
I20260925 13:17:45.240487 5792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08dc25c7670b4b05822c2c02b87b9e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46041 } }
I20260925 13:17:45.240520 5789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:45.240798 5792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08dc25c7670b4b05822c2c02b87b9e88. Latest consensus state: current_term: 1 leader_uuid: "08dc25c7670b4b05822c2c02b87b9e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08dc25c7670b4b05822c2c02b87b9e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46041 } } }
I20260925 13:17:45.240928 5792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:45.241014 5793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08dc25c7670b4b05822c2c02b87b9e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08dc25c7670b4b05822c2c02b87b9e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46041 } } }
I20260925 13:17:45.241108 5793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88 [sys.catalog]: This master's current role is: LEADER
W20260925 13:17:45.242194 5809 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:17:45.242238 5809 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:17:45.242334 5801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:45.242486 5801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:45.243952 5801 catalog_manager.cc:1347] Generated new cluster ID: 6792d0344970472fa27d8ccd6d60f6be
I20260925 13:17:45.244005 5801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:17:45.246447 5797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:45.240659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.0.98.62:46041)
0925 13:17:45.241023 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:45.241032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:45.241071 (+ 39us) server_negotiation.cc:407] Connection header received
0925 13:17:45.241210 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:45.241218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:45.241298 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:45.241442 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:45.241718 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.243153 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.244146 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.244841 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.244970 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:45.245743 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:45.245747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:45.245747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:45.245947 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:45.245986 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:45.246001 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:45.246060 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:45.246151 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 13:17:45.246229 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":80,"threads_started":1}
I20260925 13:17:45.247346 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:17:45.268744 5801 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:45.269306 5801 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:45.276988 5801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08dc25c7670b4b05822c2c02b87b9e88: Generated new TSK 0
I20260925 13:17:45.277180 5801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:17:45.287539 5797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:45.284022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54374 (local address 127.0.98.62:46041)
0925 13:17:45.284068 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:45.284070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:45.284152 (+ 82us) server_negotiation.cc:407] Connection header received
0925 13:17:45.284235 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:45.284236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:45.284246 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:45.284304 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0925 13:17:45.284435 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.284913 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.286098 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.286292 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.286340 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:45.286513 (+ 173us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 13:17:45.287329 (+ 816us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40829/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 13:17:45.287455 (+ 126us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54374: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40829/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 13:17:45.287611 5797 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54374: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40829/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 13:17:45.287611 5818 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:45.283928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46041 (local address 127.0.0.1:54374)
0925 13:17:45.284081 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:45.284093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:45.284161 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:45.284315 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:45.284316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:45.284318 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 13:17:45.284379 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:45.284382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.284998 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.284999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:45.286018 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:45.286023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.286355 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.286359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:45.286419 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:45.286426 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 13:17:45.287419 (+ 993us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40829/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 13:17:45.287520 (+ 101us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46041: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40829/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260925 13:17:45.287767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 5722
E20260925 13:17:45.299953 5819 webserver.cc:533] Webserver: sslize failed:
E20260925 13:17:45.300045 5819 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T13:17:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (207 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T13:17:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:45Z Disabled control of system clock
I20260925 13:17:45.332252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40781
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40781
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/testchainca.cert with env {}
W20260925 13:17:45.411059 5823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:45.411301 5823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:45.411324 5823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:45.412480 5823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:45.412518 5823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:45.412529 5823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:45.412540 5823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:45.412551 5823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 13:17:45.412562 5823 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 13:17:45.412572 5823 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 13:17:45.414083 5823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:45.414294 5823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:45.414470 5823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:45.419286 5831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:45.419339 5829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:45.419281 5828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:45.419477 5823 server_base.cc:1034] running on GCE node
I20260925 13:17:45.419777 5823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:45.420076 5823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:45.421253 5823 hybrid_clock.cc:648] HybridClock initialized: now 1790342265421232 us; error 35 us; skew 500 ppm
I20260925 13:17:45.422611 5823 webserver.cc:460] Webserver started at http://127.0.98.62:43131/ using document root <none> and password file <none>
I20260925 13:17:45.422829 5823 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:45.422904 5823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:45.423032 5823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:45.424125 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "6b7a6821732246e098bb0f8f776a727e"
format_stamp: "Formatted at 2026-09-25 13:17:45 on dist-test-slave-t8d9"
server_key: "3b5184cec55b4173f9fe230208a4f9d4"
server_key_iv: "171c189b12f06f2a1abfef49d14a28a0"
server_key_version: "encryptionkey@0"
I20260925 13:17:45.424458 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "6b7a6821732246e098bb0f8f776a727e"
format_stamp: "Formatted at 2026-09-25 13:17:45 on dist-test-slave-t8d9"
server_key: "3b5184cec55b4173f9fe230208a4f9d4"
server_key_iv: "171c189b12f06f2a1abfef49d14a28a0"
server_key_version: "encryptionkey@0"
I20260925 13:17:45.425725 5823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:45.426414 5837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:45.426546 5823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:45.426630 5823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "6b7a6821732246e098bb0f8f776a727e"
format_stamp: "Formatted at 2026-09-25 13:17:45 on dist-test-slave-t8d9"
server_key: "3b5184cec55b4173f9fe230208a4f9d4"
server_key_iv: "171c189b12f06f2a1abfef49d14a28a0"
server_key_version: "encryptionkey@0"
I20260925 13:17:45.426692 5823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:45.449519 5823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:45.449909 5823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:45.450067 5823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:45.453703 5823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40781
I20260925 13:17:45.453742 5889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40781 every 8 connection(s)
I20260925 13:17:45.454082 5823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:45.454973 5890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:45.456378 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 5823
I20260925 13:17:45.456496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:45.456676 392 external_mini_cluster.cc:1428] Setting key 117baee4ef716b59d3d40928228ed3fe
I20260925 13:17:45.457653 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e: Bootstrap starting.
I20260925 13:17:45.458298 5890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:45.458581 5890 log.cc:824] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:45.459223 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e: No bootstrap required, opened a new log
I20260925 13:17:45.461470 5890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7a6821732246e098bb0f8f776a727e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40781 } }
I20260925 13:17:45.461696 5890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:45.461726 5890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7a6821732246e098bb0f8f776a727e, State: Initialized, Role: FOLLOWER
I20260925 13:17:45.461849 5890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7a6821732246e098bb0f8f776a727e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40781 } }
I20260925 13:17:45.461920 5890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:45.461941 5890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:45.461993 5890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:45.462456 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:45.457200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.0.98.62:40781)
0925 13:17:45.457383 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:45.457390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:45.457416 (+ 26us) server_negotiation.cc:407] Connection header received
0925 13:17:45.457917 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:45.457925 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:45.457996 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:45.458126 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:45.458341 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.459310 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.460394 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.460807 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.460905 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:45.461661 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:45.461665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:45.461666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:17:45.461929 (+ 263us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:45.461986 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:17:45.462003 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:45.462078 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:45.462157 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 13:17:45.462230 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260925 13:17:45.462617 5890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7a6821732246e098bb0f8f776a727e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40781 } }
I20260925 13:17:45.462723 5890 leader_election.cc:302] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b7a6821732246e098bb0f8f776a727e; no voters:
I20260925 13:17:45.462893 5890 leader_election.cc:288] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:45.462934 5895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:45.463079 5895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [term 1 LEADER]: Becoming Leader. State: Replica: 6b7a6821732246e098bb0f8f776a727e, State: Running, Role: LEADER
I20260925 13:17:45.463137 5890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:45.463227 5895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7a6821732246e098bb0f8f776a727e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40781 } }
I20260925 13:17:45.463560 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:17:45.463615 5897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b7a6821732246e098bb0f8f776a727e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7a6821732246e098bb0f8f776a727e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40781 } } }
I20260925 13:17:45.463685 5897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:45.463814 5897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b7a6821732246e098bb0f8f776a727e. Latest consensus state: current_term: 1 leader_uuid: "6b7a6821732246e098bb0f8f776a727e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7a6821732246e098bb0f8f776a727e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40781 } } }
I20260925 13:17:45.463853 5897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:45.463975 5904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:45.464505 5904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:45.466113 5904 catalog_manager.cc:1347] Generated new cluster ID: 7563204ccb404f3595816ddee1799b7a
I20260925 13:17:45.466176 5904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:17:45.477507 5904 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:45.478262 5904 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:45.486227 5904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b7a6821732246e098bb0f8f776a727e: Generated new TSK 0
I20260925 13:17:45.486474 5904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:17:45.605891 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:45.494679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.0.98.62:40781)
0925 13:17:45.494725 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:45.494727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:45.494804 (+ 77us) server_negotiation.cc:407] Connection header received
0925 13:17:45.494885 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:45.494885 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:45.494896 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:45.494955 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0925 13:17:45.495077 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.495582 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.496705 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.496897 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.496943 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:45.497113 (+ 170us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 13:17:45.605651 (+108538us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 13:17:45.605806 (+ 155us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40258: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 13:17:45.605962 5893 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40258: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 13:17:45.606021 5919 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:45.494576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40781 (local address 127.0.0.1:40258)
0925 13:17:45.494732 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:45.494742 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:45.494815 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:45.494965 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:45.494966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:45.494968 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 13:17:45.495027 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:45.495029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.495670 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.495671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:45.496616 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:45.496619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:45.496962 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:45.496965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:45.497025 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:45.497032 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 13:17:45.605795 (+108763us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 13:17:45.605923 (+ 128us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40781: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260925 13:17:45.606335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 5823
2026-09-25T13:17:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (297 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (88 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:17:45 dist-test-slave-t8d9 krb5kdc[5931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:17:45 dist-test-slave-t8d9 krb5kdc[5931](info): set up 2 sockets
Sep 25 13:17:45 dist-test-slave-t8d9 krb5kdc[5931](info): commencing operation
krb5kdc: starting...
W20260925 13:17:46.725610 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:17:46 dist-test-slave-t8d9 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:17:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:17:46.837385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:46.917317 5944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:46.917593 5944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:46.917618 5944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:46.918895 5944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:46.918954 5944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:46.918968 5944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:46.918980 5944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:46.918991 5944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:46.920676 5944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:46.920949 5944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:46.921167 5944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:46.925897 5952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:46.925979 5950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:46.925912 5949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:46.926080 5944 server_base.cc:1034] running on GCE node
I20260925 13:17:46.926433 5944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:46.926704 5944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:46.927840 5944 hybrid_clock.cc:648] HybridClock initialized: now 1790342266927824 us; error 32 us; skew 500 ppm
Sep 25 13:17:46 dist-test-slave-t8d9 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:46.930194 5944 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:17:46.930631 5944 webserver.cc:460] Webserver started at http://127.0.98.62:41217/ using document root <none> and password file <none>
I20260925 13:17:46.930838 5944 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:46.930886 5944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:46.931018 5944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:46.932073 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "56f03436a1cd49d98dd91a3996c7355d"
format_stamp: "Formatted at 2026-09-25 13:17:46 on dist-test-slave-t8d9"
server_key: "3a1664055f63831fe2db063617c2e5ed"
server_key_iv: "fbdb01441bab0dbb192b7ab8baf4fecc"
server_key_version: "encryptionkey@0"
I20260925 13:17:46.932389 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "56f03436a1cd49d98dd91a3996c7355d"
format_stamp: "Formatted at 2026-09-25 13:17:46 on dist-test-slave-t8d9"
server_key: "3a1664055f63831fe2db063617c2e5ed"
server_key_iv: "fbdb01441bab0dbb192b7ab8baf4fecc"
server_key_version: "encryptionkey@0"
I20260925 13:17:46.933674 5944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:17:46.934414 5959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:46.934599 5944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:46.934682 5944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "56f03436a1cd49d98dd91a3996c7355d"
format_stamp: "Formatted at 2026-09-25 13:17:46 on dist-test-slave-t8d9"
server_key: "3a1664055f63831fe2db063617c2e5ed"
server_key_iv: "fbdb01441bab0dbb192b7ab8baf4fecc"
server_key_version: "encryptionkey@0"
I20260925 13:17:46.934749 5944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:46.946725 5944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:46.948961 5944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:46.949106 5944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:46.952872 5944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35889
I20260925 13:17:46.952919 6011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35889 every 8 connection(s)
I20260925 13:17:46.953419 5944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:46.954146 6012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:46.956895 6012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d: Bootstrap starting.
I20260925 13:17:46.957474 6012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:46.957759 6012 log.cc:824] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:46.958427 6012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d: No bootstrap required, opened a new log
I20260925 13:17:46.960019 6012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f03436a1cd49d98dd91a3996c7355d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35889 } }
I20260925 13:17:46.960168 6012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:46.960201 6012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56f03436a1cd49d98dd91a3996c7355d, State: Initialized, Role: FOLLOWER
I20260925 13:17:46.960351 6012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f03436a1cd49d98dd91a3996c7355d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35889 } }
I20260925 13:17:46.960436 6012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:46.960471 6012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:46.960516 6012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:46.961403 6012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f03436a1cd49d98dd91a3996c7355d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35889 } }
I20260925 13:17:46.961496 6012 leader_election.cc:302] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56f03436a1cd49d98dd91a3996c7355d; no voters:
I20260925 13:17:46.961709 6012 leader_election.cc:288] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:46.961751 6015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:46.961800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 5944
I20260925 13:17:46.961906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:46.961908 6015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [term 1 LEADER]: Becoming Leader. State: Replica: 56f03436a1cd49d98dd91a3996c7355d, State: Running, Role: LEADER
I20260925 13:17:46.961975 6012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:46.961982 6015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f03436a1cd49d98dd91a3996c7355d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35889 } }
I20260925 13:17:46.962085 392 external_mini_cluster.cc:1428] Setting key 103c4e2f7549a935c8f12c1c3de8cfc7
I20260925 13:17:46.962296 6015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56f03436a1cd49d98dd91a3996c7355d. Latest consensus state: current_term: 1 leader_uuid: "56f03436a1cd49d98dd91a3996c7355d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f03436a1cd49d98dd91a3996c7355d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35889 } } }
I20260925 13:17:46.962363 6015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:46.962637 6019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:46.963194 6019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:46.963745 6016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56f03436a1cd49d98dd91a3996c7355d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f03436a1cd49d98dd91a3996c7355d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35889 } } }
I20260925 13:17:46.963825 6016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:46.965102 6019 catalog_manager.cc:1347] Generated new cluster ID: 27b6895d7f0c4dd3a5264260d4298434
I20260925 13:17:46.965152 6019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:17:46 dist-test-slave-t8d9 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:46.973335 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:46.962596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43918 (local address 127.0.98.62:35889)
0925 13:17:46.962759 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:46.962770 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:46.962861 (+ 91us) server_negotiation.cc:407] Connection header received
0925 13:17:46.963049 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:46.963057 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:46.963116 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:46.964031 (+ 915us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:46.964124 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:46.965560 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:46.966602 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:46.967221 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:46.969595 (+ 2374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:46.969872 (+ 277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:46.969876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:46.969877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:46.969887 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:46.971766 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:46.972138 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:46.972141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:46.972142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:46.972248 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:46.972410 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:46.972410 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:46.972411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:46.972717 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:46.972865 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:46.972939 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 13:17:46.973080 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260925 13:17:46.984907 6019 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:46.985437 6019 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:46.990370 6019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56f03436a1cd49d98dd91a3996c7355d: Generated new TSK 0
I20260925 13:17:46.990571 6019 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:17:47.017825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35889
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:47.098553 6038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:47.098762 6038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:47.098794 6038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:47.099954 6038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:47.099993 6038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:47.100028 6038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:17:47.101482 6038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:47.101733 6038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:47.101938 6038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:47.106549 6043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:47.106659 6038 server_base.cc:1034] running on GCE node
W20260925 13:17:47.106580 6044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:47.106565 6046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:47.106945 6038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:47.107155 6038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:47.108281 6038 hybrid_clock.cc:648] HybridClock initialized: now 1790342267108273 us; error 26 us; skew 500 ppm
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:47.110772 6038 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:17:47.111217 6038 webserver.cc:460] Webserver started at http://127.0.98.1:42095/ using document root <none> and password file <none>
I20260925 13:17:47.111418 6038 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:47.111459 6038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:47.111591 6038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:47.112648 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "85556a24fff446eab565ea65ba685457"
format_stamp: "Formatted at 2026-09-25 13:17:47 on dist-test-slave-t8d9"
server_key: "0e1f6e3e2049d2a7a90930096c4a5309"
server_key_iv: "85521364396726782a8f32481dab40c2"
server_key_version: "encryptionkey@0"
I20260925 13:17:47.112973 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "85556a24fff446eab565ea65ba685457"
format_stamp: "Formatted at 2026-09-25 13:17:47 on dist-test-slave-t8d9"
server_key: "0e1f6e3e2049d2a7a90930096c4a5309"
server_key_iv: "85521364396726782a8f32481dab40c2"
server_key_version: "encryptionkey@0"
I20260925 13:17:47.114522 6038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:17:47.115180 6053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:47.115329 6038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:47.115409 6038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "85556a24fff446eab565ea65ba685457"
format_stamp: "Formatted at 2026-09-25 13:17:47 on dist-test-slave-t8d9"
server_key: "0e1f6e3e2049d2a7a90930096c4a5309"
server_key_iv: "85521364396726782a8f32481dab40c2"
server_key_version: "encryptionkey@0"
I20260925 13:17:47.115473 6038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:47.123944 6038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:47.126017 6038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:47.126145 6038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:47.126338 6038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:47.126626 6038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:47.126673 6038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:47.126703 6038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:47.126720 6038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:47.132257 6038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40365
I20260925 13:17:47.132296 6166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40365 every 8 connection(s)
I20260925 13:17:47.132611 6038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:47.141234 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:47.133311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45165 (local address 127.0.98.62:35889)
0925 13:17:47.133385 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:47.133387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:47.133578 (+ 191us) server_negotiation.cc:407] Connection header received
0925 13:17:47.134513 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:47.134514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:47.134527 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:47.134576 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:47.135148 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.135653 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.136568 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.136750 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.136795 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:47.138969 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:47.138971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:47.138971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:47.138977 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:47.140112 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:47.140454 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:47.140455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:47.140455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:47.140507 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:47.140685 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:47.140686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:47.140686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:47.140786 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:47.140828 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:47.141074 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 13:17:47.141163 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 13:17:47.141351 6169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:47.133283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35889 (local address 127.0.98.1:45165)
0925 13:17:47.133436 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:47.133474 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:47.134418 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:47.134635 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:47.134639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:47.134890 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:47.135086 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:47.135090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.135717 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.135718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:47.136511 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:47.136512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.136805 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.136806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:47.136902 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:47.137417 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:47.137428 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:47.138884 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:47.140174 (+ 1290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:47.140180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:47.140182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:47.140389 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:47.140555 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:47.140556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:47.140556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:47.140645 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:47.140825 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:47.140827 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:47.140946 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 13:17:47.141038 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 13:17:47.141163 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260925 13:17:47.141868 6167 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35889
I20260925 13:17:47.141971 6167 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:47.142122 6167 heartbeater.cc:511] Master 127.0.98.62:35889 requested a full tablet report, sending...
I20260925 13:17:47.142607 5976 ts_manager.cc:194] Registered new tserver with Master: 85556a24fff446eab565ea65ba685457 (127.0.98.1:40365)
I20260925 13:17:47.142725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 6038
I20260925 13:17:47.142819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:17:47.143008 392 external_mini_cluster.cc:1428] Setting key 243544140a63f88d83231a2346607923
I20260925 13:17:47.143364 5976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45165
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:17:47.168407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35889
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:47.248991 6172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:47.249208 6172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:47.249231 6172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:47.250458 6172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:47.250510 6172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:47.250536 6172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:17:47.252080 6172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:47.252336 6172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:47.252560 6172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:47.257431 6177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:47.257440 6178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:47.257457 6180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:47.257682 6172 server_base.cc:1034] running on GCE node
I20260925 13:17:47.257911 6172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:47.258193 6172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:47.259351 6172 hybrid_clock.cc:648] HybridClock initialized: now 1790342267259331 us; error 38 us; skew 500 ppm
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:47.261863 6172 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:17:47.262326 6172 webserver.cc:460] Webserver started at http://127.0.98.2:44019/ using document root <none> and password file <none>
I20260925 13:17:47.262564 6172 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:47.262610 6172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:47.262737 6172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:47.263875 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "d745958ac4334c86b149504ccf7a6a05"
format_stamp: "Formatted at 2026-09-25 13:17:47 on dist-test-slave-t8d9"
server_key: "42f11e94768d9c503cae76980c37fca7"
server_key_iv: "13da0712ed6b8b71d1acb3999120afe6"
server_key_version: "encryptionkey@0"
I20260925 13:17:47.264200 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d745958ac4334c86b149504ccf7a6a05"
format_stamp: "Formatted at 2026-09-25 13:17:47 on dist-test-slave-t8d9"
server_key: "42f11e94768d9c503cae76980c37fca7"
server_key_iv: "13da0712ed6b8b71d1acb3999120afe6"
server_key_version: "encryptionkey@0"
I20260925 13:17:47.265431 6172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:17:47.266204 6187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:47.266366 6172 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:17:47.266448 6172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "d745958ac4334c86b149504ccf7a6a05"
format_stamp: "Formatted at 2026-09-25 13:17:47 on dist-test-slave-t8d9"
server_key: "42f11e94768d9c503cae76980c37fca7"
server_key_iv: "13da0712ed6b8b71d1acb3999120afe6"
server_key_version: "encryptionkey@0"
I20260925 13:17:47.266515 6172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:47.275316 6172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:47.277311 6172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:47.277451 6172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:47.277750 6172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:47.278045 6172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:47.278077 6172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:47.278105 6172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:47.278129 6172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:47.283512 6172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41901
I20260925 13:17:47.283583 6300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41901 every 8 connection(s)
I20260925 13:17:47.283892 6172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:47.292604 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:47.284619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48099 (local address 127.0.98.62:35889)
0925 13:17:47.284675 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:47.284677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:47.284845 (+ 168us) server_negotiation.cc:407] Connection header received
0925 13:17:47.285841 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:47.285842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:47.285855 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:47.285904 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:47.286520 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.287022 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.287987 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.288183 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.288239 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:47.290271 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:47.290273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:47.290273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:47.290280 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:47.291353 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:47.291705 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:47.291706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:47.291707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:47.291757 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:47.291963 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:47.291964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:47.291964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:47.292072 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:47.292128 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:47.292419 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 13:17:47.292512 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 13:17:47.292784 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:47.284591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35889 (local address 127.0.98.2:48099)
0925 13:17:47.284732 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:47.284767 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:47.285745 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:47.285972 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:47.285975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:47.286235 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:47.286424 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:47.286429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.287103 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.287104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:47.287906 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:47.287908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.288250 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.288250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:47.288364 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:47.288833 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:47.288842 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:47.290189 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:47.291420 (+ 1231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:47.291427 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:47.291429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:47.291641 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:47.291809 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:47.291810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:47.291810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:47.291904 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:47.292125 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:47.292132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:47.292292 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 13:17:47.292402 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 13:17:47.292567 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260925 13:17:47.293232 6301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35889
I20260925 13:17:47.293335 6301 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:47.293325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 6172
I20260925 13:17:47.293412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:17:47.293560 6301 heartbeater.cc:511] Master 127.0.98.62:35889 requested a full tablet report, sending...
I20260925 13:17:47.293624 392 external_mini_cluster.cc:1428] Setting key 68db34be5ca7b67a16845cb2261dd68d
I20260925 13:17:47.293927 5975 ts_manager.cc:194] Registered new tserver with Master: d745958ac4334c86b149504ccf7a6a05 (127.0.98.2:41901)
I20260925 13:17:47.294361 5975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48099
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:17:47.316900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35889
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:47.398788 6306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:47.399019 6306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:47.399042 6306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:47.400277 6306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:47.400336 6306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:47.400364 6306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:17:47.401991 6306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:47.402242 6306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:47.402464 6306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:47.407302 6314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:47.407330 6312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:47.407330 6311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:47.407564 6306 server_base.cc:1034] running on GCE node
I20260925 13:17:47.407857 6306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:47.408103 6306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:47.409247 6306 hybrid_clock.cc:648] HybridClock initialized: now 1790342267409232 us; error 30 us; skew 500 ppm
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:47.411819 6306 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:17:47.412251 6306 webserver.cc:460] Webserver started at http://127.0.98.3:42971/ using document root <none> and password file <none>
I20260925 13:17:47.412475 6306 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:47.412524 6306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:47.412648 6306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:47.413728 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "309c4aa1e9de459ca0e9334b901681d2"
format_stamp: "Formatted at 2026-09-25 13:17:47 on dist-test-slave-t8d9"
server_key: "99191efd324be7de6d1451e14495ed72"
server_key_iv: "72aae7eabc049d30f5988b09e3d6422d"
server_key_version: "encryptionkey@0"
I20260925 13:17:47.414052 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "309c4aa1e9de459ca0e9334b901681d2"
format_stamp: "Formatted at 2026-09-25 13:17:47 on dist-test-slave-t8d9"
server_key: "99191efd324be7de6d1451e14495ed72"
server_key_iv: "72aae7eabc049d30f5988b09e3d6422d"
server_key_version: "encryptionkey@0"
I20260925 13:17:47.415256 6306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:47.416426 6321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:47.416649 6306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:47.416744 6306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "309c4aa1e9de459ca0e9334b901681d2"
format_stamp: "Formatted at 2026-09-25 13:17:47 on dist-test-slave-t8d9"
server_key: "99191efd324be7de6d1451e14495ed72"
server_key_iv: "72aae7eabc049d30f5988b09e3d6422d"
server_key_version: "encryptionkey@0"
I20260925 13:17:47.416821 6306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:47.438107 6306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:47.440169 6306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:47.440303 6306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:47.440526 6306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:47.440822 6306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:47.440855 6306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:47.440884 6306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:47.440908 6306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:47.446331 6306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38025
I20260925 13:17:47.446379 6434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38025 every 8 connection(s)
I20260925 13:17:47.446683 6306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:17:47.451989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 6306
I20260925 13:17:47.452091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:17:47.452271 392 external_mini_cluster.cc:1428] Setting key b33334d71861cdf4473e7bcb6ebfc758
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:47.455142 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:47.447300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49725 (local address 127.0.98.62:35889)
0925 13:17:47.447359 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:47.447360 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:47.447555 (+ 195us) server_negotiation.cc:407] Connection header received
0925 13:17:47.448470 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:47.448471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:47.448480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:47.448535 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:47.449113 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.449643 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.450627 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.450799 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.450838 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:47.452933 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:47.452934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:47.452934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:47.452941 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:47.454109 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:47.454457 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:47.454458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:47.454458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:47.454501 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:47.454675 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:47.454675 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:47.454675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:47.454769 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:47.454812 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:47.455026 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 13:17:47.455080 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:17:47.455343 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:47.447286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35889 (local address 127.0.98.3:49725)
0925 13:17:47.447438 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:47.447480 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:47.448389 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:47.448593 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:47.448596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:47.448851 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:47.449047 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:47.449052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.449700 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.449701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:47.450562 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:47.450564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.450847 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.450848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:47.450955 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:47.451350 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:47.451358 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:47.452852 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:47.454178 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:47.454185 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:47.454186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:47.454392 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:47.454544 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:47.454544 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:47.454544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:47.454634 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:47.454808 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:47.454811 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:47.454923 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 13:17:47.455014 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 13:17:47.455131 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260925 13:17:47.455747 6435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35889
I20260925 13:17:47.455829 6435 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:47.456001 6435 heartbeater.cc:511] Master 127.0.98.62:35889 requested a full tablet report, sending...
I20260925 13:17:47.456323 5975 ts_manager.cc:194] Registered new tserver with Master: 309c4aa1e9de459ca0e9334b901681d2 (127.0.98.3:38025)
I20260925 13:17:47.456663 5975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49725
I20260925 13:17:47.464118 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/test-admin.keytab.
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:47.477447 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 13:17:47.478938 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:47.478303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.0.98.1:40365)
0925 13:17:47.478468 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:47.478473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:47.478483 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:17:47.478508 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:47.478510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:47.478524 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:47.478587 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:47.478862 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60810: BlockingRecv error: recv got EOF from 127.0.0.1:60810 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
W20260925 13:17:47.479044 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:47.479948 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:47.480752 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:17:47.487705 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:47.481258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60824 (local address 127.0.98.1:40365)
0925 13:17:47.481312 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:47.481314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:47.481323 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:47.481380 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:47.481381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:47.481391 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:47.481437 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:47.481833 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.482566 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.483210 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.483521 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.483582 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:47.485459 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:47.485469 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:47.485469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:47.485476 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:47.486776 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:47.487081 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:47.487084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:47.487084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:47.487134 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:47.487258 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:47.487258 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:47.487258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:47.487390 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:47.487486 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:47.487544 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 13:17:47.487602 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 13:17:47.489275 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:47.488832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.0.98.1:40365)
0925 13:17:47.488872 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:47.488873 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:47.488923 (+ 50us) server_negotiation.cc:407] Connection header received
0925 13:17:47.488971 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:47.488972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:47.488980 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:47.489023 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:47.489236 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60838: BlockingRecv error: recv got EOF from 127.0.0.1:60838 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 13:17:47.489427 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/krb5cc)
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:47.490388 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:47.491204 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:17:47.497136 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:47.491666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60844 (local address 127.0.98.1:40365)
0925 13:17:47.491712 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:47.491714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:47.491755 (+ 41us) server_negotiation.cc:407] Connection header received
0925 13:17:47.491791 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:47.491792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:47.491800 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:47.491840 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:47.492134 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.492605 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.493223 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.493401 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.493441 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:47.495206 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:47.495207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:47.495207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:47.495213 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:47.496383 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:47.496626 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:47.496627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:47.496627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:47.496671 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:47.496783 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:47.496783 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:47.496783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:47.496863 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:47.496923 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:47.496994 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 13:17:47.497050 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 13:17:47.498469 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:47.498049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.0.98.1:40365)
0925 13:17:47.498085 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:47.498086 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:47.498120 (+ 34us) server_negotiation.cc:407] Connection header received
0925 13:17:47.498173 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:47.498174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:47.498183 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:47.498230 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:47.498423 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60858: BlockingRecv error: recv got EOF from 127.0.0.1:60858 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 13:17:47.498533 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790342239834770-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:47.499370 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:47.500145 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:17:47.506217 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:47.500637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60868 (local address 127.0.98.1:40365)
0925 13:17:47.500685 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:47.500687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:47.500705 (+ 18us) server_negotiation.cc:407] Connection header received
0925 13:17:47.500746 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:47.500747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:47.500755 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:47.500790 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:47.501089 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.501588 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.502257 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:47.502427 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:47.502466 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:47.504246 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:47.504247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:47.504247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:47.504254 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:47.505422 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:47.505705 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:47.505706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:47.505706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:47.505753 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:47.505876 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:47.505877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:47.505877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:47.505952 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:47.506017 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:47.506080 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 13:17:47.506131 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 13:17:47.506790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 6038
I20260925 13:17:47.511157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 6172
I20260925 13:17:47.514916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 6306
I20260925 13:17:47.518783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 5944
2026-09-25T13:17:47Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1738 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[6455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[6455](info): set up 2 sockets
Sep 25 13:17:47 dist-test-slave-t8d9 krb5kdc[6455](info): commencing operation
krb5kdc: starting...
W20260925 13:17:48.481566 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:17:48 dist-test-slave-t8d9 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:17:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 13:17:48.596393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34171
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:48.689623 6468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:48.689854 6468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:48.689878 6468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:48.691164 6468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:48.691210 6468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:48.691222 6468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:48.691234 6468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:48.691246 6468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:48.692754 6468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:48.692997 6468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:48.693215 6468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:48.697764 6476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:48.697772 6473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:48.697772 6474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:48.698047 6468 server_base.cc:1034] running on GCE node
I20260925 13:17:48.698266 6468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:48.698571 6468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:48.699726 6468 hybrid_clock.cc:648] HybridClock initialized: now 1790342268699708 us; error 32 us; skew 500 ppm
Sep 25 13:17:48 dist-test-slave-t8d9 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342268, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:48.702272 6468 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 13:17:48.702730 6468 webserver.cc:460] Webserver started at http://127.0.98.62:44807/ using document root <none> and password file <none>
I20260925 13:17:48.702934 6468 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:48.702978 6468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:48.703110 6468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:48.704205 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "0ced1fac976141779f951abf99ce6599"
format_stamp: "Formatted at 2026-09-25 13:17:48 on dist-test-slave-t8d9"
server_key: "d242f58ad75751032a2deb4ea4b5a031"
server_key_iv: "ce4f220abe05536d42c96c8707d8c099"
server_key_version: "encryptionkey@0"
I20260925 13:17:48.704542 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ced1fac976141779f951abf99ce6599"
format_stamp: "Formatted at 2026-09-25 13:17:48 on dist-test-slave-t8d9"
server_key: "d242f58ad75751032a2deb4ea4b5a031"
server_key_iv: "ce4f220abe05536d42c96c8707d8c099"
server_key_version: "encryptionkey@0"
I20260925 13:17:48.705843 6468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 13:17:48.706576 6483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:48.706741 6468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:48.706830 6468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "0ced1fac976141779f951abf99ce6599"
format_stamp: "Formatted at 2026-09-25 13:17:48 on dist-test-slave-t8d9"
server_key: "d242f58ad75751032a2deb4ea4b5a031"
server_key_iv: "ce4f220abe05536d42c96c8707d8c099"
server_key_version: "encryptionkey@0"
I20260925 13:17:48.706884 6468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:48.722174 6468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:48.724398 6468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:48.724535 6468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:48.728644 6468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34171
I20260925 13:17:48.728675 6535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34171 every 8 connection(s)
I20260925 13:17:48.729039 6468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:48.730088 6536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:48.731325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 6468
I20260925 13:17:48.731455 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:48.731652 392 external_mini_cluster.cc:1428] Setting key f868dfa0fd7d7b290007c1648e9f8a1b
I20260925 13:17:48.732844 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599: Bootstrap starting.
I20260925 13:17:48.733438 6536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:48.733737 6536 log.cc:824] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:48.734489 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599: No bootstrap required, opened a new log
I20260925 13:17:48.736622 6536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260925 13:17:48.736789 6536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:48.736824 6536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ced1fac976141779f951abf99ce6599, State: Initialized, Role: FOLLOWER
I20260925 13:17:48.736971 6536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260925 13:17:48.737059 6536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:48.737103 6536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:48.737150 6536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:48.737756 6536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260925 13:17:48.737846 6536 leader_election.cc:302] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ced1fac976141779f951abf99ce6599; no voters:
I20260925 13:17:48.738027 6536 leader_election.cc:288] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:48.738047 6541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:48.738150 6541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [term 1 LEADER]: Becoming Leader. State: Replica: 0ced1fac976141779f951abf99ce6599, State: Running, Role: LEADER
I20260925 13:17:48.738230 6541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260925 13:17:48.738296 6536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 13:17:48 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:17:48.738538 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } }
I20260925 13:17:48.738595 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:48.738711 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ced1fac976141779f951abf99ce6599. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } }
I20260925 13:17:48.738762 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:48.739199 6545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:48.739687 6545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:48.741176 6545 catalog_manager.cc:1347] Generated new cluster ID: a126f6acf3074ab08732f62e7f3ce19a
I20260925 13:17:48.741227 6545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:17:48.745134 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:48.732213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.0.98.62:34171)
0925 13:17:48.732379 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:48.732387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:48.732414 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:17:48.732822 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:48.732829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:48.732880 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:48.733014 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:48.733752 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:48.735046 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:48.735766 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:48.736428 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:48.736541 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:48.739091 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:48.739096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:48.739097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:48.739109 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:48.743173 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:48.743570 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:48.743573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:48.743574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:48.743681 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:48.743896 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:48.743897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:48.743898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:48.744204 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:48.744334 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:48.744797 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 13:17:48.744927 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260925 13:17:48.757148 6545 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:48.758258 6545 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:48.761440 6545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599: Generated new TSK 0
I20260925 13:17:48.761675 6545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 13:17:48.788416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34171
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:48.869089 6562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:48.869314 6562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:48.869339 6562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:48.870599 6562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:48.870651 6562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:48.870679 6562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:17:48.872200 6562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:48.872450 6562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:48.872684 6562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:17:48.877523 6562 server_base.cc:1034] running on GCE node
W20260925 13:17:48.877506 6570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:48.877508 6567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:48.877508 6568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:48.877904 6562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:48.878114 6562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:48.879245 6562 hybrid_clock.cc:648] HybridClock initialized: now 1790342268879224 us; error 41 us; skew 500 ppm
Sep 25 13:17:48 dist-test-slave-t8d9 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342268, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:48.881512 6562 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 13:17:48.881958 6562 webserver.cc:460] Webserver started at http://127.0.98.1:38587/ using document root <none> and password file <none>
I20260925 13:17:48.882198 6562 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:48.882249 6562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:48.882385 6562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:48.883713 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "0fb1305a6a8a403cb15b4b556c796b04"
format_stamp: "Formatted at 2026-09-25 13:17:48 on dist-test-slave-t8d9"
server_key: "fe989b950314a0d553cd987696dbb08d"
server_key_iv: "24f425447d7be543df3cd78dce520d0f"
server_key_version: "encryptionkey@0"
I20260925 13:17:48.884120 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0fb1305a6a8a403cb15b4b556c796b04"
format_stamp: "Formatted at 2026-09-25 13:17:48 on dist-test-slave-t8d9"
server_key: "fe989b950314a0d553cd987696dbb08d"
server_key_iv: "24f425447d7be543df3cd78dce520d0f"
server_key_version: "encryptionkey@0"
I20260925 13:17:48.885561 6562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:17:48.886245 6577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:48.886399 6562 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:48.886478 6562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "0fb1305a6a8a403cb15b4b556c796b04"
format_stamp: "Formatted at 2026-09-25 13:17:48 on dist-test-slave-t8d9"
server_key: "fe989b950314a0d553cd987696dbb08d"
server_key_iv: "24f425447d7be543df3cd78dce520d0f"
server_key_version: "encryptionkey@0"
I20260925 13:17:48.886544 6562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:48.897097 6562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:48.899132 6562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:48.899272 6562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:48.899482 6562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:48.899770 6562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:48.899801 6562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:48.899830 6562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:48.899854 6562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:48.905390 6562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35095
I20260925 13:17:48.905436 6690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35095 every 8 connection(s)
I20260925 13:17:48.905807 6562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 13:17:48 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342268, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:17:48.912951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 6562
I20260925 13:17:48.913048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:17:48.913264 392 external_mini_cluster.cc:1428] Setting key d4b2b1bf293e8aff79e7b25cbcf19aa7
I20260925 13:17:48.914942 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:48.906507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60641 (local address 127.0.98.62:34171)
0925 13:17:48.906578 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:48.906580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:48.906741 (+ 161us) server_negotiation.cc:407] Connection header received
0925 13:17:48.907236 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:48.907237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:48.907246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:48.907290 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:48.908230 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:48.908730 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:48.909782 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:48.909973 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:48.910018 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:48.912152 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:48.912154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:48.912155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:48.912161 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:48.913618 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:48.914086 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:48.914087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:48.914087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:48.914149 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:48.914363 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:48.914364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:48.914364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:48.914490 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:48.914553 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:48.914773 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 13:17:48.914853 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 13:17:48.915048 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:48.906479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:60641)
0925 13:17:48.906633 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:48.906672 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:48.907157 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:48.907355 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:48.907359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:48.907952 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:48.908159 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:48.908164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:48.908795 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:48.908795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:48.909711 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:48.909714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:48.910039 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:48.910040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:48.910142 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:48.910629 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:48.910638 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:48.912068 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:48.913693 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:48.913702 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:48.913704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:48.914011 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:48.914206 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:48.914207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:48.914207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:48.914312 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:48.914542 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:48.914545 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:48.914665 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 13:17:48.914742 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 13:17:48.914864 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260925 13:17:48.915596 6691 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34171
I20260925 13:17:48.915696 6691 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:48.915887 6691 heartbeater.cc:511] Master 127.0.98.62:34171 requested a full tablet report, sending...
I20260925 13:17:48.916384 6500 ts_manager.cc:194] Registered new tserver with Master: 0fb1305a6a8a403cb15b4b556c796b04 (127.0.98.1:35095)
I20260925 13:17:48.917213 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60641
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 13:17:48.936369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34171
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:49.015703 6696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:49.015923 6696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:49.015947 6696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:49.017141 6696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:49.017194 6696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:49.017220 6696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:17:49.018725 6696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:49.018961 6696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:49.019188 6696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:49.023978 6702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:49.023967 6704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:49.023962 6701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:49.023998 6696 server_base.cc:1034] running on GCE node
I20260925 13:17:49.024338 6696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:49.024580 6696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:49.025751 6696 hybrid_clock.cc:648] HybridClock initialized: now 1790342269025729 us; error 29 us; skew 500 ppm
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:49.028383 6696 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 13:17:49.028817 6696 webserver.cc:460] Webserver started at http://127.0.98.2:34075/ using document root <none> and password file <none>
I20260925 13:17:49.029042 6696 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:49.029090 6696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:49.029217 6696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:49.030318 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "62cf9c8eb1f74bdaa5e251667e493263"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "a67bf9f747b9de212ea528b0aab8270c"
server_key_iv: "b7965c581c038d6c2846fa1c312e87df"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.030649 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "62cf9c8eb1f74bdaa5e251667e493263"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "a67bf9f747b9de212ea528b0aab8270c"
server_key_iv: "b7965c581c038d6c2846fa1c312e87df"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.031927 6696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 13:17:49.032613 6711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.032761 6696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:17:49.032845 6696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "62cf9c8eb1f74bdaa5e251667e493263"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "a67bf9f747b9de212ea528b0aab8270c"
server_key_iv: "b7965c581c038d6c2846fa1c312e87df"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.032907 6696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:49.047709 6696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:49.049896 6696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:49.050034 6696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:49.050246 6696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:49.050545 6696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:49.050578 6696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.050606 6696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:49.050624 6696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.056188 6696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45815
I20260925 13:17:49.056231 6824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45815 every 8 connection(s)
I20260925 13:17:49.056547 6696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:17:49.061122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 6696
I20260925 13:17:49.061228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:17:49.061398 392 external_mini_cluster.cc:1428] Setting key 8c51d3dd6d93f40b048f029a80920d26
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:17:49.066100 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.057246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57743 (local address 127.0.98.62:34171)
0925 13:17:49.057301 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.057303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.057506 (+ 203us) server_negotiation.cc:407] Connection header received
0925 13:17:49.057999 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.058000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.058008 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.058065 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.058991 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.059501 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.060534 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.060707 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.060751 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.063307 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.063309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.063309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.063319 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.064645 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.065116 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.065117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.065117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.065165 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.065430 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.065430 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.065430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.065575 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.065636 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.065970 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.066033 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:17:49.066378 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.057232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:57743)
0925 13:17:49.057386 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.057430 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.057920 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.058129 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.058132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.058701 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.058907 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.058913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.059605 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.059606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.060463 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.060466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.060792 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.060793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.060961 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.061563 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.061572 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.063193 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.064726 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.064734 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.064736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.065040 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.065219 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.065219 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.065220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.065367 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.065635 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.065640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.065819 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.065953 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.066132 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260925 13:17:49.066857 6825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34171
I20260925 13:17:49.066942 6825 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:49.067116 6825 heartbeater.cc:511] Master 127.0.98.62:34171 requested a full tablet report, sending...
I20260925 13:17:49.067487 6500 ts_manager.cc:194] Registered new tserver with Master: 62cf9c8eb1f74bdaa5e251667e493263 (127.0.98.2:45815)
I20260925 13:17:49.067842 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57743
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 13:17:49.086709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34171
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:49.167353 6830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:49.167593 6830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:49.167618 6830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:49.168804 6830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:49.168853 6830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:49.168880 6830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:17:49.170382 6830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:49.170594 6830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:49.170804 6830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:49.175602 6838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:49.175679 6830 server_base.cc:1034] running on GCE node
W20260925 13:17:49.175577 6836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:49.175590 6835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:49.175988 6830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:49.176205 6830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:49.177345 6830 hybrid_clock.cc:648] HybridClock initialized: now 1790342269177328 us; error 28 us; skew 500 ppm
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:49.179801 6830 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 13:17:49.180267 6830 webserver.cc:460] Webserver started at http://127.0.98.3:42107/ using document root <none> and password file <none>
I20260925 13:17:49.180480 6830 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:49.180528 6830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:49.180650 6830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:49.181787 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "a0f8a13eb91545bfa682c88cef4201a2"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "65316004c03189bbe7a47b47be24bfee"
server_key_iv: "d0a8c22ee77e47024deda5a93db115db"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.182119 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a0f8a13eb91545bfa682c88cef4201a2"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "65316004c03189bbe7a47b47be24bfee"
server_key_iv: "d0a8c22ee77e47024deda5a93db115db"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.183374 6830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 13:17:49.184067 6845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.184208 6830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.184288 6830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "a0f8a13eb91545bfa682c88cef4201a2"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "65316004c03189bbe7a47b47be24bfee"
server_key_iv: "d0a8c22ee77e47024deda5a93db115db"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.184346 6830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:49.202353 6830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:49.204561 6830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:49.204703 6830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:49.204931 6830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:49.205247 6830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:49.205281 6830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.205310 6830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:49.205333 6830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.211124 6830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39831
I20260925 13:17:49.211170 6958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39831 every 8 connection(s)
I20260925 13:17:49.211525 6830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:17:49.220861 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.212235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52519 (local address 127.0.98.62:34171)
0925 13:17:49.212296 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.212297 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.212459 (+ 162us) server_negotiation.cc:407] Connection header received
0925 13:17:49.212977 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.212979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.212991 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.213061 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.214193 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.214709 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.215759 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.215978 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.216037 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.218239 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.218240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.218241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.218250 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.219490 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.219967 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.219968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.219968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.220030 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.220257 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.220258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.220258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.220364 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.220418 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.220718 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.220787 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:17:49.221117 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.212174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:52519)
0925 13:17:49.212326 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.212380 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.212868 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.213132 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.213136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.213874 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.214097 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.214104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.214814 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.214815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.215668 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.215671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.216059 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.216060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.216170 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.216698 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.216707 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.218149 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.219587 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.219594 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.219596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.219827 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.220100 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.220101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.220101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.220210 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.220444 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.220449 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.220609 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.220704 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.220875 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260925 13:17:49.221637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 6830
I20260925 13:17:49.221657 6959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34171
I20260925 13:17:49.221729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:17:49.221746 6959 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:49.221905 392 external_mini_cluster.cc:1428] Setting key 4f1b4a2eea1ba391cd8e516d940e95c4
I20260925 13:17:49.221957 6959 heartbeater.cc:511] Master 127.0.98.62:34171 requested a full tablet report, sending...
I20260925 13:17:49.222409 6500 ts_manager.cc:194] Registered new tserver with Master: a0f8a13eb91545bfa682c88cef4201a2 (127.0.98.3:39831)
I20260925 13:17:49.222687 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:17:49.222837 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52519
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260925 13:17:49.248198 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44113
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34171,127.0.98.61:44113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:49.328101 6964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:49.328363 6964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:49.328398 6964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:49.329614 6964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:49.329654 6964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:49.329670 6964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:49.329686 6964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:49.329701 6964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:49.331175 6964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34171,127.0.98.61:44113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
W20260925 13:17:49.331286 6964 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:34171,127.0.98.61:44113'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 13:17:49.331449 6964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:49.331650 6964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:49.336265 6970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:49.336248 6972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:49.336253 6969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:49.336292 6964 server_base.cc:1034] running on GCE node
I20260925 13:17:49.336651 6964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:49.336884 6964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:49.338085 6964 hybrid_clock.cc:648] HybridClock initialized: now 1790342269338073 us; error 29 us; skew 500 ppm
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:49.340611 6964 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 13:17:49.341033 6964 webserver.cc:460] Webserver started at http://127.0.98.61:42163/ using document root <none> and password file <none>
I20260925 13:17:49.341238 6964 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:49.341284 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:49.341383 6964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:49.342509 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/data/instance:
uuid: "9318f44ac5814b20b91da5f65f9d1f20"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "a467d7e9810f2bb0919cf4b57bcf2c92"
server_key_iv: "33682e9e174ee2998be773c40639cc2c"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.342813 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/wal/instance:
uuid: "9318f44ac5814b20b91da5f65f9d1f20"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "a467d7e9810f2bb0919cf4b57bcf2c92"
server_key_iv: "33682e9e174ee2998be773c40639cc2c"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.344087 6964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:17:49.344798 6979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.345017 6964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:49.345093 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/wal
uuid: "9318f44ac5814b20b91da5f65f9d1f20"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "a467d7e9810f2bb0919cf4b57bcf2c92"
server_key_iv: "33682e9e174ee2998be773c40639cc2c"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.345173 6964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:49.355374 6964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:49.357235 6964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:49.357368 6964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:49.360870 6964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44113
I20260925 13:17:49.360920 7031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44113 every 8 connection(s)
I20260925 13:17:49.361234 6964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/data/info.pb
I20260925 13:17:49.362218 7032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:49.362840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 6964
I20260925 13:17:49.363941 7032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 13:17:49.372952 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.364696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.0.98.62:34171)
0925 13:17:49.364743 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.364745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.364893 (+ 148us) server_negotiation.cc:407] Connection header received
0925 13:17:49.364928 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.364930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.364941 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.365013 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.365429 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.366188 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.367399 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.367671 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.367724 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.369972 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.369973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.369974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.369982 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.371485 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.371891 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.371892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.371893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.371954 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.372229 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.372230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.372230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.372429 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.372493 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.372771 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.372837 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 13:17:49.373136 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.364603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.0.1:51996)
0925 13:17:49.364762 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.364807 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.364878 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.365035 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.365037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.365265 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.365353 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.365358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.366275 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.366276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.367334 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.367336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.367734 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.367735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.367801 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.368299 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.368309 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.369885 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.371578 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.371584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.371585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.371823 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.372019 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.372019 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.372020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.372136 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.372486 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.372490 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.372644 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.372744 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.372900 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260925 13:17:49.373286 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.363472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.0.98.61:44113)
0925 13:17:49.363650 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.363659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.363682 (+ 23us) server_negotiation.cc:407] Connection header received
0925 13:17:49.364084 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.364090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.364136 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.364258 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.364939 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.366211 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.367232 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.367611 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.367705 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.370465 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.370474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.370474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.370481 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.372121 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.372474 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.372478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.372478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.372535 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.372691 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.372692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.372693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.372854 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.372984 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.373109 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.373183 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260925 13:17:49.373720 7032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 13:17:49.378969 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.373944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44113 (local address 127.0.0.1:59974)
0925 13:17:49.373982 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.373992 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.374026 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.374172 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.374173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.374299 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.374353 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.374355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.374915 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.374916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.375392 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.375393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.375671 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.375672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.375717 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.376073 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.376078 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.377108 (+ 1030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.378276 (+ 1168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.378277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.378277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.378422 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.378560 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.378560 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.378560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.378599 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.378746 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.378746 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.378776 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.378818 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.378888 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 13:17:49.379020 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.373986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.0.98.61:44113)
0925 13:17:49.374016 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.374018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.374025 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:17:49.374069 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.374070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.374079 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.374133 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.374405 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.374850 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.375443 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.375624 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.375671 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.377185 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.377186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.377186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.377192 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.378217 (+ 1025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.378480 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.378481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.378481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.378519 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.378630 (+ 111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.378630 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.378630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.378709 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.378747 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.378832 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.378925 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":14}
I20260925 13:17:49.380218 7032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: Bootstrap starting.
I20260925 13:17:49.380951 7032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:49.381160 7032 log.cc:824] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:49.381912 7032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: No bootstrap required, opened a new log
I20260925 13:17:49.383311 7032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } }
I20260925 13:17:49.383446 7032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:49.383468 7032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9318f44ac5814b20b91da5f65f9d1f20, State: Initialized, Role: FOLLOWER
I20260925 13:17:49.383559 7032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } }
I20260925 13:17:49.383801 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } } }
I20260925 13:17:49.383867 7032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:49.383904 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 13:17:49.385375 7049 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:17:49.385430 7049 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:17:49.386837 6964 master_runner.cc:428] Detected that this master 9318f44ac5814b20b91da5f65f9d1f20 is joining an existing cluster
I20260925 13:17:49.386883 6964 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:44113
I20260925 13:17:49.393414 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.388033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44113 (local address 127.0.0.1:59980)
0925 13:17:49.388126 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.388134 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.388168 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.388276 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.388276 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.388487 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.388534 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.388536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.389178 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.389179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.389871 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.389873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.390190 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.390190 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.390242 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.390631 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.390636 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.390913 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.392427 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.392428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.392428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.392606 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.392801 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.392802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.392802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.392854 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.393084 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.393085 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.393131 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.393198 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.393293 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":62,"threads_started":1}
I20260925 13:17:49.393416 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.388174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.0.98.61:44113)
0925 13:17:49.388202 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.388204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.388210 (+ 6us) server_negotiation.cc:407] Connection header received
0925 13:17:49.388224 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.388225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.388232 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.388274 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.388599 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.389113 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.389939 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.390127 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.390175 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.390993 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.390994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.390994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.391000 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.392365 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.392676 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.392677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.392677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.392738 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.392913 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.392914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.392914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.393029 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.393092 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.393222 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.393340 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":16}
I20260925 13:17:49.394868 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.387816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.0.98.62:34171)
0925 13:17:49.387878 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.387881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.388025 (+ 144us) server_negotiation.cc:407] Connection header received
0925 13:17:49.388062 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.388063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.388075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.388137 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.388640 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.389394 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.390517 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.390801 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.390851 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.391852 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.391853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.391853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.391860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.393426 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.394020 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.394021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.394022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.394099 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.394311 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.394312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.394313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.394462 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.394527 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.394719 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.394791 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 13:17:49.395144 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.387756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.0.1:51998)
0925 13:17:49.387951 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.387961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.388013 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.388189 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.388191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.388453 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.388541 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.388544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.389487 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.389488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.390417 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.390420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.390925 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.390926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.390993 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.391345 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.391349 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.391741 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.393727 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.393727 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.393728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.393934 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.394157 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.394157 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.394158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.394235 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.394556 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.394557 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.394611 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.394951 (+ 340us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.395052 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":90,"threads_started":1}
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342268, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 13:17:49.403861 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.396941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44113 (local address 127.0.0.1:59988)
0925 13:17:49.397113 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.397161 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.397236 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.397362 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.397365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.397717 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.397772 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.397774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.398338 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.398338 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.399263 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.399266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.399568 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.399569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.399644 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.400021 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.400025 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.401417 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.402921 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.402928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.402930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.403122 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.403299 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.403300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.403300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.403346 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.403579 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.403582 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.403624 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.403683 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.403771 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":113,"threads_started":1}
I20260925 13:17:49.403909 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.396987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59988 (local address 127.0.98.61:44113)
0925 13:17:49.397027 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.397028 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.397221 (+ 193us) server_negotiation.cc:407] Connection header received
0925 13:17:49.397306 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.397307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.397314 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.397354 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.397835 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.398275 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.399331 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.399514 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.399556 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.401498 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.401500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.401500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.401507 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.402848 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.403185 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.403186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.403186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.403242 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.403391 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.403392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.403392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.403516 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.403593 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.403728 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.403803 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:17:49.404486 6500 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:44113: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 44113
}
attrs {
promote: true
}
}
dest_uuid: "0ced1fac976141779f951abf99ce6599"
cas_config_opid_index: -1
I20260925 13:17:49.404783 6500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: true } }
I20260925 13:17:49.405256 7059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9318f44ac5814b20b91da5f65f9d1f20 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: true } } }
I20260925 13:17:49.405617 7006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:49.406322 7006 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 FOLLOWER]: Refusing update from remote peer 0ced1fac976141779f951abf99ce6599: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 13:17:49.406342 7059 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:44113
I20260925 13:17:49.406424 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: true } } }
I20260925 13:17:49.406477 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:49.406560 6964 master.cc:537] Master@127.0.98.61:44113 shutting down...
I20260925 13:17:49.406744 7058 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 13:17:49.407397 6486 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 13:17:49.407755 6486 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 -> Peer 9318f44ac5814b20b91da5f65f9d1f20 (127.0.98.61:44113): Couldn't send request to peer 9318f44ac5814b20b91da5f65f9d1f20. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 13:17:49.408903 6964 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:49.408990 6964 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:49.409039 6964 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: stopping tablet replica
I20260925 13:17:49.421690 6964 master.cc:552] Master@127.0.98.61:44113 shutdown complete.
I20260925 13:17:49.421742 6964 master_runner.cc:305] Clearing existing system tablet
I20260925 13:17:49.421855 6964 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:49.421897 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:49.422449 6964 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:49.423131 7063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.423309 6964 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:49.423358 6964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:17:49.423382 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/wal
uuid: "9318f44ac5814b20b91da5f65f9d1f20"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "a467d7e9810f2bb0919cf4b57bcf2c92"
server_key_iv: "33682e9e174ee2998be773c40639cc2c"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.424055 6964 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:49.425516 6964 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 13:17:49.425621 6964 log.cc:1196] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 13:17:49.425786 6964 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: Deleting consensus metadata
I20260925 13:17:49.425997 6964 master_runner.cc:315] Copying system tablet from 127.0.98.62:34171
I20260925 13:17:49.426584 6964 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34171
I20260925 13:17:49.432119 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.426976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.0.98.62:34171)
0925 13:17:49.427027 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.427029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.427094 (+ 65us) server_negotiation.cc:407] Connection header received
0925 13:17:49.427162 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.427163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.427173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.427218 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.427509 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.428010 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.428741 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.428936 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.428998 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.429946 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.429948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.429948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.429956 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.431170 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.431453 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.431454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.431454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.431507 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.431658 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.431659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.431659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.431768 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.431823 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.431949 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.432021 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 13:17:49.432133 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.426914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.0.1:52006)
0925 13:17:49.427029 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.427040 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.427111 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.427222 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.427222 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.427389 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.427438 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.427440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.428083 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.428084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.428660 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.428662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.428996 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.428997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.429056 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.429573 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.429578 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.429867 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.431240 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.431241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.431241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.431396 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.431556 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.431557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.431557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.431611 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.431818 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.431819 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.431861 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.431921 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.432021 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260925 13:17:49.432660 6520 tablet_copy_service.cc:140] P 0ced1fac976141779f951abf99ce6599: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9318f44ac5814b20b91da5f65f9d1f20 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:52006)
I20260925 13:17:49.432775 6520 tablet_copy_service.cc:161] P 0ced1fac976141779f951abf99ce6599: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9318f44ac5814b20b91da5f65f9d1f20 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:52006: session id = 9318f44ac5814b20b91da5f65f9d1f20-00000000000000000000000000000000
I20260925 13:17:49.433617 6520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 13:17:49.434257 6964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 13:17:49.435278 6964 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: tablet copy: Starting download of 0 data blocks...
I20260925 13:17:49.435417 6964 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: tablet copy: Starting download of 1 WAL segments...
I20260925 13:17:49.436373 6964 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 13:17:49.437191 6520 tablet_copy_service.cc:342] P 0ced1fac976141779f951abf99ce6599: Request end of tablet copy session 9318f44ac5814b20b91da5f65f9d1f20-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:52006
I20260925 13:17:49.437256 6520 tablet_copy_service.cc:433] P 0ced1fac976141779f951abf99ce6599: ending tablet copy session 9318f44ac5814b20b91da5f65f9d1f20-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9318f44ac5814b20b91da5f65f9d1f20
I20260925 13:17:49.440150 6964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:49.441291 7070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:49.441380 7071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:49.441387 7073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:49.441674 6964 server_base.cc:1034] running on GCE node
I20260925 13:17:49.441762 6964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:49.441885 6964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:49.442991 6964 hybrid_clock.cc:648] HybridClock initialized: now 1790342269442986 us; error 25 us; skew 500 ppm
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:49.444316 6964 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 13:17:49.444576 6964 webserver.cc:460] Webserver started at http://127.0.98.61:36979/ using document root <none> and password file <none>
I20260925 13:17:49.444687 6964 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:49.444721 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:49.445426 6964 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:49.445991 7080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.446094 6964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.446135 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/wal
uuid: "9318f44ac5814b20b91da5f65f9d1f20"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "a467d7e9810f2bb0919cf4b57bcf2c92"
server_key_iv: "33682e9e174ee2998be773c40639cc2c"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.446185 6964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:49.455094 6964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:49.456486 6964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:49.458808 6964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44113
I20260925 13:17:49.459120 7132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44113 every 8 connection(s)
I20260925 13:17:49.459156 6964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-1/data/info.pb
I20260925 13:17:49.459982 7133 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:17:49.460505 7133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: Bootstrap starting.
I20260925 13:17:49.463011 7133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:17:49.463222 7133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: Bootstrap complete.
I20260925 13:17:49.463408 7133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: true } }
I20260925 13:17:49.463487 7133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9318f44ac5814b20b91da5f65f9d1f20, State: Initialized, Role: LEARNER
I20260925 13:17:49.463524 7133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: true } }
I20260925 13:17:49.463698 7133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:49.463714 7136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: true } } }
I20260925 13:17:49.463843 7136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: This master's current role is: LEARNER
I20260925 13:17:49.464911 7147 catalog_manager.cc:1259] Loaded cluster ID: a126f6acf3074ab08732f62e7f3ce19a
I20260925 13:17:49.464957 7147 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: loading cluster ID for follower catalog manager: success
I20260925 13:17:49.466094 7147 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: acquiring CA information for follower catalog manager: success
I20260925 13:17:49.466404 7147 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 13:17:49.480880 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.475103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.0.98.61:44113)
0925 13:17:49.475237 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.475238 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.475245 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:17:49.475265 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.475265 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.475274 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.475329 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.475864 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.476504 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.477166 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.477358 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.477411 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.478557 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.478558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.478558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.478566 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.479881 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.480186 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.480187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.480188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.480238 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.480392 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.480393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.480393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.480503 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.480587 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.480693 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.480771 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260925 13:17:49.510488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44189
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34171,127.0.98.61:44113,127.0.98.60:44189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:49.591869 7151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:49.592093 7151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:49.592124 7151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:49.593284 7151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:49.593324 7151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:49.593343 7151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:49.593359 7151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:49.593375 7151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:49.594964 7151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34171,127.0.98.61:44113,127.0.98.60:44189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:49.595218 7151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:49.595430 7151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:49.600119 7156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:49.600342 7151 server_base.cc:1034] running on GCE node
W20260925 13:17:49.600226 7159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:49.600155 7157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:49.600718 7151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:49.601017 7151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:49.602154 7151 hybrid_clock.cc:648] HybridClock initialized: now 1790342269602141 us; error 28 us; skew 500 ppm
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:49.604657 7151 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 13:17:49.605074 7151 webserver.cc:460] Webserver started at http://127.0.98.60:35149/ using document root <none> and password file <none>
I20260925 13:17:49.605268 7151 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:49.605305 7151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:49.605396 7151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:49.606443 7151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/data/instance:
uuid: "ee3b3de7a7ca49a4b136c62850c91065"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "705e00b502d1b7482261d07ebca64cb5"
server_key_iv: "d43f730bf7c9ca480e4387011d1b4271"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.606738 7151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/wal/instance:
uuid: "ee3b3de7a7ca49a4b136c62850c91065"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "705e00b502d1b7482261d07ebca64cb5"
server_key_iv: "d43f730bf7c9ca480e4387011d1b4271"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.607963 7151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:17:49.608606 7166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.608752 7151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:49.608836 7151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/wal
uuid: "ee3b3de7a7ca49a4b136c62850c91065"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "705e00b502d1b7482261d07ebca64cb5"
server_key_iv: "d43f730bf7c9ca480e4387011d1b4271"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.608903 7151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:49.631084 7151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:49.633214 7151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:49.633337 7151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:49.637400 7151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44189
I20260925 13:17:49.637441 7218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44189 every 8 connection(s)
I20260925 13:17:49.637794 7151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/data/info.pb
I20260925 13:17:49.638753 7219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:49.640324 7219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:17:49.645671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 7151
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:17:49.649360 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.641106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.0.98.62:34171)
0925 13:17:49.641162 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.641163 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.641218 (+ 55us) server_negotiation.cc:407] Connection header received
0925 13:17:49.641700 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.641701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.641712 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.641771 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.642377 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.642865 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.643839 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.644032 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.644086 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.646583 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.646585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.646586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.646594 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.648023 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.648396 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.648397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.648397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.648464 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.648725 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.648726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.648726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.648858 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.648907 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.649203 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.649267 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 13:17:49.649647 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.640960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.0.1:52010)
0925 13:17:49.641115 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.641157 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.641600 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.641831 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.641834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.642092 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.642287 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.642291 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.642935 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.642935 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.643765 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.643767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.644094 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.644095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.644194 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.644662 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.644670 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.646497 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.648106 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.648112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.648114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.648331 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.648546 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.648547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.648548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.648666 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.648911 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.648915 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.649077 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.649190 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.649358 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260925 13:17:49.650207 7219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 13:17:49.654376 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.646199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.0.98.60:44189)
0925 13:17:49.646346 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.646350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.646368 (+ 18us) server_negotiation.cc:407] Connection header received
0925 13:17:49.646388 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.646390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.646409 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.646485 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.647156 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.647976 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.648824 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.649235 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.649293 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.651745 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.651753 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.651754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.651761 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.653259 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.653615 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.653619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.653619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.653675 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.653848 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.653849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.653850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.654013 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.654165 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.654209 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.654277 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260925 13:17:49.656421 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.650447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44113 (local address 127.0.0.1:60016)
0925 13:17:49.650489 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.650499 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.650533 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.650668 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.650668 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.650798 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.650851 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.650853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.651566 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.651567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.652132 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.652134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.652481 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.652481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.652550 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.652823 (+ 273us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.652826 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.654002 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.655574 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.655575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.655576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.655722 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.655880 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.655881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.655881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.655933 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.656136 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.656136 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.656176 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.656236 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.656323 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 13:17:49.656422 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.650523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60016 (local address 127.0.98.61:44113)
0925 13:17:49.650573 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.650574 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.650583 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:49.650603 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.650604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.650616 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.650661 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.650928 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.651472 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.652203 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.652417 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.652471 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.654076 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.654078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.654078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.654085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.655504 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.655780 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.655781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.655781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.655831 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.655982 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.655983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.655983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.656087 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.656138 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.656263 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.656336 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:17:49.656845 7219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 13:17:49.662233 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.657065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:34954)
0925 13:17:49.657106 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.657117 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.657153 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.657247 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.657248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.657344 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.657373 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.657374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.657949 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.657949 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.658441 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.658442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.658721 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.658721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.658754 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.659100 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.659104 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.660155 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.661447 (+ 1292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.661448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.661448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.661638 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.661774 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.661774 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.661775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.661817 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.661998 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.661998 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.662031 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.662078 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.662149 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 13:17:49.662285 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.657122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.0.98.60:44189)
0925 13:17:49.657155 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.657157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.657163 (+ 6us) server_negotiation.cc:407] Connection header received
0925 13:17:49.657196 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.657197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.657204 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.657242 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.657416 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.657896 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.658499 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.658676 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.658720 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.660228 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.660230 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.660230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.660236 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.661372 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.661691 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.661691 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.661692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.661733 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.661860 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.661860 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.661860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.661954 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.661994 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.662096 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.662191 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":17}
I20260925 13:17:49.663530 7219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: Bootstrap starting.
I20260925 13:17:49.664348 7219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:49.664577 7219 log.cc:824] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:49.665323 7219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: No bootstrap required, opened a new log
I20260925 13:17:49.666757 7219 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } }
I20260925 13:17:49.666896 7219 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:49.666918 7219 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee3b3de7a7ca49a4b136c62850c91065, State: Initialized, Role: FOLLOWER
I20260925 13:17:49.667024 7219 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } }
I20260925 13:17:49.667299 7219 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:49.667261 7224 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } } }
I20260925 13:17:49.667342 7224 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 13:17:49.668644 7235 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:17:49.668699 7235 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:17:49.671028 7151 master_runner.cc:428] Detected that this master ee3b3de7a7ca49a4b136c62850c91065 is joining an existing cluster
I20260925 13:17:49.671075 7151 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:44189
I20260925 13:17:49.678833 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.671969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.0.98.62:34171)
0925 13:17:49.672013 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.672015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.672200 (+ 185us) server_negotiation.cc:407] Connection header received
0925 13:17:49.672259 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.672260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.672272 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.672322 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.672748 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.673596 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.674811 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.675128 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.675191 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.676424 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.676426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.676426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.676434 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.677730 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.678117 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.678118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.678118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.678173 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.678358 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.678358 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.678359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.678464 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.678509 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.678671 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.678743 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:17:49.679162 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.671932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.0.1:52014)
0925 13:17:49.672134 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.672145 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.672200 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.672368 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.672369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.672588 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.672665 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.672668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.673692 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.673693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.674711 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.674714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.675268 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.675269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.675342 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.675775 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.675780 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.676324 (+ 544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.677800 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.677802 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.677802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.678037 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.678233 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.678234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.678234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.678302 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.678516 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.678517 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.678577 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.678650 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.678762 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":43,"thread_start_us":149,"threads_started":1}
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342268, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 13:17:49.704901 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.681159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:34972)
0925 13:17:49.681203 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.681213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.681249 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.681515 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.681517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.681728 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.681782 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.681784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.682474 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.682476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.683330 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.683333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.683707 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.683708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.683781 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.684593 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.684599 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.686212 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.703809 (+ 17597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.703811 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.703811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.704051 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.704256 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.704257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.704257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.704310 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.704591 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.704592 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.704642 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.704687 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.704803 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 13:17:49.705808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 6562
I20260925 13:17:49.705921 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.681222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.0.98.60:44189)
0925 13:17:49.681394 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.681396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.681401 (+ 5us) server_negotiation.cc:407] Connection header received
0925 13:17:49.681430 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.681430 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.681439 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.681492 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.681885 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.682401 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.683413 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.683638 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.683694 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.686344 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.686346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.686346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.686353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.703684 (+ 17331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.704134 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.704135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.704135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.704188 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.704388 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.704388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.704389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.704522 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.705053 (+ 531us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.705097 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.705794 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":113,"threads_started":1}
I20260925 13:17:49.706499 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.672383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.0.98.61:44113)
0925 13:17:49.672434 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.672436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.672444 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:17:49.672463 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.672464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.672473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.672515 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.672847 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.673646 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.674914 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.675215 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.675250 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.703211 (+ 27961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.703213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.703213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.703222 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.704765 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.705673 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.705674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.705675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.705741 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.705957 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.705958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.705958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.706088 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.706127 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.706313 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.706411 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 13:17:49.706784 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.671963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44113 (local address 127.0.0.1:60024)
0925 13:17:49.672284 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.672292 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.672325 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.672556 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.672556 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.672700 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.672763 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.672764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.673731 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.673732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.674834 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.674836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.675312 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.675313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.675363 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.702198 (+ 26835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.702207 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.703060 (+ 853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.704859 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.704860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.704860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.705351 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.705814 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.705814 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.705815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.705888 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.706165 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.706166 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.706219 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.706576 (+ 357us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.706684 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":42,"thread_start_us":114,"threads_started":1}
I20260925 13:17:49.710899 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.672199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:34960)
0925 13:17:49.672765 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.672773 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.672837 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.672966 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.672966 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.673140 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.673194 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.673196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.674048 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.674049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.674823 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.674825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.675294 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.675295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.675354 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.702442 (+ 27088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.702448 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.703200 (+ 752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.705483 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.705483 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.705484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.705716 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.710185 (+ 4469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.710186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.710186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.710268 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.710548 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.710549 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.710600 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.710663 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.710792 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":68,"threads_started":1}
I20260925 13:17:49.710999 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.672428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34960 (local address 127.0.98.60:44189)
0925 13:17:49.672463 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.672465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.672810 (+ 345us) server_negotiation.cc:407] Connection header received
0925 13:17:49.672889 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.672890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.672897 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.672918 (+ 21us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.673274 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.673974 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.674904 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.675192 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.675228 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.703277 (+ 28049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.703277 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.703278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.703283 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.705418 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.709993 (+ 4575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.709994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.709995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.710106 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.710334 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.710335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.710336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.710487 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.710571 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.710687 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.710875 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":17}
I20260925 13:17:49.711928 6500 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:44189: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 44189
}
attrs {
promote: true
}
}
dest_uuid: "0ced1fac976141779f951abf99ce6599"
cas_config_opid_index: 5
I20260925 13:17:49.712183 6500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: true } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } }
I20260925 13:17:49.712548 7059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ee3b3de7a7ca49a4b136c62850c91065 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: true } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } } }
I20260925 13:17:49.713269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35095
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34171
--webserver_port=38587
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34171,127.0.98.61:44113
--tserver_master_addrs=127.0.98.62:34171,127.0.98.61:44113,127.0.98.60:44189 with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:17:49.713452 7059 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:44189
I20260925 13:17:49.713522 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: true } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } } }
I20260925 13:17:49.713611 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:49.713827 7151 master.cc:537] Master@127.0.98.60:44189 shutting down...
W20260925 13:17:49.716391 6486 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 -> Peer ee3b3de7a7ca49a4b136c62850c91065 (127.0.98.60:44189): Couldn't send request to peer ee3b3de7a7ca49a4b136c62850c91065. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 13:17:49.717037 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.712640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44113 (local address 127.0.0.1:60030)
0925 13:17:49.712826 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.712837 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.712887 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.713099 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.713101 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.713102 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:49.713148 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.713150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.714450 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.714451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.715688 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.715692 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.716766 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.716767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.716835 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.716864 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.716914 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.716962 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260925 13:17:49.717237 7151 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:49.717331 7151 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:49.717372 7151 tablet_replica.cc:330] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: stopping tablet replica
I20260925 13:17:49.717510 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.712734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60030 (local address 127.0.98.61:44113)
0925 13:17:49.712798 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.712800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.712935 (+ 135us) server_negotiation.cc:407] Connection header received
0925 13:17:49.712959 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.712960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.712983 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.713038 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:49.713213 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.714012 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.715851 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.716463 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.716533 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.716617 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.717397 (+ 780us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.717444 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 13:17:49.717724 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 LEARNER]: Refusing update from remote peer 0ced1fac976141779f951abf99ce6599: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 13:17:49.717909 7058 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 13:17:49.718592 7107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ee3b3de7a7ca49a4b136c62850c91065 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: true } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } } }
I20260925 13:17:49.719866 7136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ced1fac976141779f951abf99ce6599. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: true } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } } }
I20260925 13:17:49.719954 7136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: This master's current role is: LEARNER
I20260925 13:17:49.719908 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: true } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } } }
I20260925 13:17:49.720000 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: This master's current role is: LEARNER
I20260925 13:17:49.721199 7058 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599: attempting to promote NON_VOTER 9318f44ac5814b20b91da5f65f9d1f20 to VOTER
I20260925 13:17:49.721438 7058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } }
I20260925 13:17:49.722256 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 LEARNER]: Refusing update from remote peer 0ced1fac976141779f951abf99ce6599: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 13:17:49.722455 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 13:17:49.722903 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.722082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:34974)
0925 13:17:49.722822 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.722871 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":720}
I20260925 13:17:49.723160 7059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9318f44ac5814b20b91da5f65f9d1f20 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } } }
I20260925 13:17:49.724061 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } } }
I20260925 13:17:49.724145 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:49.724186 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } } }
I20260925 13:17:49.724231 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:49.724632 7107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9318f44ac5814b20b91da5f65f9d1f20 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } } }
I20260925 13:17:49.725576 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } } }
I20260925 13:17:49.725713 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:17:49.725713 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.724446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:34976)
0925 13:17:49.725645 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.725680 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1177}
W20260925 13:17:49.725772 6486 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 -> Peer ee3b3de7a7ca49a4b136c62850c91065 (127.0.98.60:44189): Couldn't send request to peer ee3b3de7a7ca49a4b136c62850c91065. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 13:17:49.732400 7151 master.cc:552] Master@127.0.98.60:44189 shutdown complete.
I20260925 13:17:49.732468 7151 master_runner.cc:305] Clearing existing system tablet
I20260925 13:17:49.732599 7151 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:49.732643 7151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:49.733582 7151 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:49.734083 7250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.734233 7151 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:49.734282 7151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.734302 7151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/wal
uuid: "ee3b3de7a7ca49a4b136c62850c91065"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "705e00b502d1b7482261d07ebca64cb5"
server_key_iv: "d43f730bf7c9ca480e4387011d1b4271"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.735092 7151 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:49.736466 7151 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 13:17:49.736531 7151 log.cc:1196] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 13:17:49.736714 7151 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: Deleting consensus metadata
I20260925 13:17:49.736816 7151 master_runner.cc:315] Copying system tablet from 127.0.98.62:34171
I20260925 13:17:49.737386 7151 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34171
I20260925 13:17:49.748485 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.737756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.0.1:52016)
0925 13:17:49.741685 (+ 3929us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.741700 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.741801 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.741976 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.741977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.742192 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.742246 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.742248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.743199 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.743200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.744067 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.744071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.744522 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.744523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.744586 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.745103 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.745108 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.745520 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.747374 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.747375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.747375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.747598 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.747793 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.747794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.747794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.747861 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.748116 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.748117 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.748176 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.748244 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.748351 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3900,"thread_start_us":78,"threads_started":1}
I20260925 13:17:49.748503 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.737817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52016 (local address 127.0.98.62:34171)
0925 13:17:49.737866 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.737868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.741795 (+ 3927us) server_negotiation.cc:407] Connection header received
0925 13:17:49.741863 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.741865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.741879 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.741958 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.742338 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.743108 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.744154 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.744447 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.744513 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.745625 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.745626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.745627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.745635 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.747292 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.747670 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.747671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.747672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.747733 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.747918 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.747918 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.747918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.748056 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.748117 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.748341 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.748413 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:17:49.748948 6520 tablet_copy_service.cc:140] P 0ced1fac976141779f951abf99ce6599: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ee3b3de7a7ca49a4b136c62850c91065 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:52016)
I20260925 13:17:49.749010 6520 tablet_copy_service.cc:161] P 0ced1fac976141779f951abf99ce6599: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ee3b3de7a7ca49a4b136c62850c91065 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:52016: session id = ee3b3de7a7ca49a4b136c62850c91065-00000000000000000000000000000000
I20260925 13:17:49.749594 6520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 13:17:49.750089 7151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 13:17:49.751439 7151 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: tablet copy: Starting download of 0 data blocks...
I20260925 13:17:49.751585 7151 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: tablet copy: Starting download of 1 WAL segments...
I20260925 13:17:49.752368 7151 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 13:17:49.753413 6520 tablet_copy_service.cc:342] P 0ced1fac976141779f951abf99ce6599: Request end of tablet copy session ee3b3de7a7ca49a4b136c62850c91065-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:52016
I20260925 13:17:49.753473 6520 tablet_copy_service.cc:433] P 0ced1fac976141779f951abf99ce6599: ending tablet copy session ee3b3de7a7ca49a4b136c62850c91065-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ee3b3de7a7ca49a4b136c62850c91065
I20260925 13:17:49.755945 7151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:49.757190 7257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:49.762631 7258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:49.762639 7260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:49.765702 7151 server_base.cc:1034] running on GCE node
I20260925 13:17:49.765856 7151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:49.766026 7151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:49.767171 7151 hybrid_clock.cc:648] HybridClock initialized: now 1790342269767150 us; error 40 us; skew 500 ppm
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:49.768781 7151 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 13:17:49.769093 7151 webserver.cc:460] Webserver started at http://127.0.98.60:42173/ using document root <none> and password file <none>
I20260925 13:17:49.769207 7151 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:49.769240 7151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:49.770058 7151 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:17:49.770617 7267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.770777 7151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:17:49.770860 7151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/wal
uuid: "ee3b3de7a7ca49a4b136c62850c91065"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "705e00b502d1b7482261d07ebca64cb5"
server_key_iv: "d43f730bf7c9ca480e4387011d1b4271"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.770922 7151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:49.801333 7151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:49.803222 7151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:49.806373 7151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44189
I20260925 13:17:49.806411 7319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44189 every 8 connection(s)
I20260925 13:17:49.806715 7151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/master-2/data/info.pb
I20260925 13:17:49.807650 7320 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:17:49.808135 7320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: Bootstrap starting.
W20260925 13:17:49.810017 7245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:49.810261 7245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:49.810305 7245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 13:17:49.811157 7320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:17:49.811362 7320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: Bootstrap complete.
I20260925 13:17:49.811554 7320 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } }
I20260925 13:17:49.811641 7320 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee3b3de7a7ca49a4b136c62850c91065, State: Initialized, Role: LEARNER
I20260925 13:17:49.811676 7320 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } }
I20260925 13:17:49.811825 7320 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:49.811833 7326 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } } }
I20260925 13:17:49.812028 7326 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [sys.catalog]: This master's current role is: LEARNER
W20260925 13:17:49.812033 7245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:49.812069 7245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:49.812121 7245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:17:49.813439 7337 catalog_manager.cc:1259] Loaded cluster ID: a126f6acf3074ab08732f62e7f3ce19a
I20260925 13:17:49.813670 7337 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: loading cluster ID for follower catalog manager: success
I20260925 13:17:49.814332 7245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38587
--tserver_master_addrs=127.0.98.62:34171,127.0.98.61:44113,127.0.98.60:44189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:49.814532 7245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:49.814710 7245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:17:49.815135 7337 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: acquiring CA information for follower catalog manager: success
I20260925 13:17:49.815516 7337 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 13:17:49.820464 7340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:49.820454 7342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:49.820482 7339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:49.821362 7245 server_base.cc:1034] running on GCE node
I20260925 13:17:49.821596 7245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:49.821835 7245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:49.822999 7245 hybrid_clock.cc:648] HybridClock initialized: now 1790342269822986 us; error 32 us; skew 500 ppm
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:49.825635 7245 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 13:17:49.826159 7245 webserver.cc:460] Webserver started at http://127.0.98.1:38587/ using document root <none> and password file <none>
I20260925 13:17:49.826413 7245 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:49.826483 7245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:49.828197 7245 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:17:49.828841 7349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.829006 7245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:49.829093 7245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "0fb1305a6a8a403cb15b4b556c796b04"
format_stamp: "Formatted at 2026-09-25 13:17:48 on dist-test-slave-t8d9"
server_key: "fe989b950314a0d553cd987696dbb08d"
server_key_iv: "24f425447d7be543df3cd78dce520d0f"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.829468 7245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:49.848587 7245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:49.850432 7245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:49.850586 7245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:49.850817 7245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:49.851131 7245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:49.851166 7245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.851197 7245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:49.851213 7245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.856805 7245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35095
I20260925 13:17:49.856875 7462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35095 every 8 connection(s)
I20260925 13:17:49.857208 7245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:17:49.859869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 7245
I20260925 13:17:49.860036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 6696
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:17:49.867936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45815
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34171
--webserver_port=34075
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34171,127.0.98.61:44113
--tserver_master_addrs=127.0.98.62:34171,127.0.98.61:44113,127.0.98.60:44189 with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:17:49.869156 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.858323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35153 (local address 127.0.98.61:44113)
0925 13:17:49.858368 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.858370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.858503 (+ 133us) server_negotiation.cc:407] Connection header received
0925 13:17:49.858912 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.858913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.858926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.858956 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.859910 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.860689 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.862354 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.862663 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.862705 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.865917 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.865918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.865918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.865927 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.867702 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.868059 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.868061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.868061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.868119 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.868380 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.868381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.868381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.868523 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.868563 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.869000 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.869067 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:17:49.869288 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.858079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.98.1:47895)
0925 13:17:49.858198 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.858242 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.858797 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.859310 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.859311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.859546 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.859805 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.859811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.860973 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.860974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.862239 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.862242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.862771 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.862772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.862915 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.863665 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.863670 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.865446 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.867577 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.867585 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.867587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.868067 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.868242 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.868242 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.868242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.868309 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.868587 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.868593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.868766 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.868870 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.869062 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":91,"thread_start_us":63,"threads_started":1}
I20260925 13:17:49.869450 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.858251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44113 (local address 127.0.98.1:35153)
0925 13:17:49.858433 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.858441 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.858814 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.859055 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.859055 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.859507 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.859803 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.859809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.860864 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.860864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.862242 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.862244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.862787 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.862788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.862915 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.863594 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.863604 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.865809 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.867786 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.867787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.867787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.867993 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.868181 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.868181 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.868182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.868310 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.868849 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.868850 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.868886 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.869270 (+ 384us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.869368 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":91,"thread_start_us":51,"threads_started":1}
I20260925 13:17:49.869607 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.858397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47895 (local address 127.0.98.60:44189)
0925 13:17:49.858521 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.858523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.858532 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:49.858877 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.858878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.858891 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.858959 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.859913 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.860878 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.862341 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.862675 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.863182 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.865586 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.865588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.865588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.865596 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.867494 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.868127 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.868128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.868129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.868183 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.868372 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.868372 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.868373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.868507 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.868563 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.869403 (+ 840us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.869484 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260925 13:17:49.869921 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44113
I20260925 13:17:49.869937 7464 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44189
I20260925 13:17:49.870035 7464 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:49.870036 7465 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:49.870196 7464 heartbeater.cc:511] Master 127.0.98.60:44189 requested a full tablet report, sending...
I20260925 13:17:49.870200 7465 heartbeater.cc:511] Master 127.0.98.61:44113 requested a full tablet report, sending...
I20260925 13:17:49.870364 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.857919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45439 (local address 127.0.98.62:34171)
0925 13:17:49.857969 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.857971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.858332 (+ 361us) server_negotiation.cc:407] Connection header received
0925 13:17:49.858913 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.858915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.858926 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.858956 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.859911 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.860697 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.862355 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.862664 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.862705 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.866366 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:49.866368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:49.866368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:49.866377 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:49.868364 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.868833 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.868834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.868834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.868896 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:49.869113 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:49.869114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:49.869114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:49.869243 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:49.869283 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:49.870200 (+ 917us) server_negotiation.cc:299] Negotiation successful
0925 13:17:49.870279 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:17:49.870700 7097 ts_manager.cc:194] Registered new tserver with Master: 0fb1305a6a8a403cb15b4b556c796b04 (127.0.98.1:35095)
I20260925 13:17:49.870703 7284 ts_manager.cc:194] Registered new tserver with Master: 0fb1305a6a8a403cb15b4b556c796b04 (127.0.98.1:35095)
I20260925 13:17:49.870827 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.857923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:45439)
0925 13:17:49.858198 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.858243 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.858796 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.859042 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.859048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.859520 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.859803 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.859809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.860816 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.860817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.862242 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.862244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:49.862788 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:49.862789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:49.862962 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:49.863614 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:49.863618 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:49.866291 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:49.868587 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.868588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.868588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.868759 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.868972 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:49.868972 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:49.868973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:49.869042 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:49.869862 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:49.869862 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:49.869911 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 13:17:49.870621 (+ 710us) client_negotiation.cc:240] Negotiation successful
0925 13:17:49.870725 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":107,"thread_start_us":90,"threads_started":1}
I20260925 13:17:49.871124 7463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34171
I20260925 13:17:49.871169 7463 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:49.871227 7463 heartbeater.cc:511] Master 127.0.98.62:34171 requested a full tablet report, sending...
I20260925 13:17:49.871446 6500 ts_manager.cc:194] Re-registered known tserver with Master: 0fb1305a6a8a403cb15b4b556c796b04 (127.0.98.1:35095)
I20260925 13:17:49.871869 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45439
W20260925 13:17:49.951712 7471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:49.951953 7471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:49.951989 7471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:49.953222 7471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:49.953269 7471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:49.953307 7471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:17:49.954917 7471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34075
--tserver_master_addrs=127.0.98.62:34171,127.0.98.61:44113,127.0.98.60:44189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:49.955183 7471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:49.955405 7471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:49.960503 7476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:49.960495 7479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:49.960525 7471 server_base.cc:1034] running on GCE node
W20260925 13:17:49.960466 7477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:49.960906 7471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:49.961153 7471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:49.962309 7471 hybrid_clock.cc:648] HybridClock initialized: now 1790342269962291 us; error 31 us; skew 500 ppm
Sep 25 13:17:49 dist-test-slave-t8d9 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:49.965006 7471 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 13:17:49.965485 7471 webserver.cc:460] Webserver started at http://127.0.98.2:34075/ using document root <none> and password file <none>
I20260925 13:17:49.965723 7471 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:49.965796 7471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:49.967145 7471 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:49.967700 7486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.967860 7471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:49.967944 7471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "62cf9c8eb1f74bdaa5e251667e493263"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "a67bf9f747b9de212ea528b0aab8270c"
server_key_iv: "b7965c581c038d6c2846fa1c312e87df"
server_key_version: "encryptionkey@0"
I20260925 13:17:49.968287 7471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:49.988420 7471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:49.990571 7471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:49.990693 7471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:49.990916 7471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:49.991221 7471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:49.991259 7471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.991288 7471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:49.991305 7471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:49.996640 7471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45815
I20260925 13:17:49.996703 7599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45815 every 8 connection(s)
I20260925 13:17:49.997053 7471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:17:50 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 13:17:50 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:17:50.004878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 7471
Sep 25 13:17:50 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342269, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 13:17:50.005048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 6830
I20260925 13:17:50.007939 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.998225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43603 (local address 127.0.98.61:44113)
0925 13:17:49.998281 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.998283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.998324 (+ 41us) server_negotiation.cc:407] Connection header received
0925 13:17:49.998805 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.998807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.998817 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.998861 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:50.000416 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.000874 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.001840 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.002045 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.002105 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.004484 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:50.004486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:50.004486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:50.004495 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:50.006287 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.006853 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.006854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.006854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.006911 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.007206 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.007207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.007208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.007346 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:50.007387 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.007762 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.007844 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:17:50.008049 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.997783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35195 (local address 127.0.98.60:44189)
0925 13:17:49.997828 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.997830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.998109 (+ 279us) server_negotiation.cc:407] Connection header received
0925 13:17:49.998756 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.998758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.998772 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.998815 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.999561 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.000320 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.001654 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.001956 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.002043 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.005397 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:50.005399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:50.005399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:50.005407 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:50.006984 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.007309 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.007310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.007311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.007360 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.007553 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.007554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.007555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.007675 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:50.007713 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.007891 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.007971 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:17:50.008303 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.997840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38645 (local address 127.0.98.62:34171)
0925 13:17:49.997889 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:49.997890 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:49.998136 (+ 246us) server_negotiation.cc:407] Connection header received
0925 13:17:49.998686 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:49.998688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:49.998701 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:49.998749 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:49.999533 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.000320 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.002499 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.002699 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.002755 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.005114 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:50.005116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:50.005116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:50.005125 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:50.007183 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.007542 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.007543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.007544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.007605 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.007774 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.007775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.007775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.007909 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:50.007947 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.008135 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.008210 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:17:50.008495 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.997793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.98.2:35195)
0925 13:17:49.998033 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.998043 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.998596 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.999195 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.999196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.999331 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.999444 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.999450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.000432 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.000433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.001563 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.001566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.002045 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.002046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.002184 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.003469 (+ 1285us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:50.003473 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:50.005298 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:50.007059 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:50.007060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:50.007060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:50.007238 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:50.007424 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:50.007425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:50.007425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:50.007488 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:50.007755 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:50.007756 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:50.007789 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 13:17:50.008115 (+ 326us) client_negotiation.cc:240] Negotiation successful
0925 13:17:50.008248 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":79,"thread_start_us":80,"threads_started":1}
I20260925 13:17:50.008495 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.998109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44113 (local address 127.0.98.2:43603)
0925 13:17:49.998277 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.998284 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.998596 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.998885 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.998886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.999249 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.999444 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.999450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.000965 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.000966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.001770 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.001772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.002120 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.002121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.002187 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.002761 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:50.002772 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:50.004383 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:50.006437 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:50.006448 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:50.006451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:50.006758 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:50.006994 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:50.006994 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:50.006995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:50.007133 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:50.007428 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:50.007434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:50.007616 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 13:17:50.008050 (+ 434us) client_negotiation.cc:240] Negotiation successful
0925 13:17:50.008312 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":122,"thread_start_us":60,"threads_started":1}
I20260925 13:17:50.008836 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:49.997793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:38645)
0925 13:17:49.997972 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:49.998015 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:49.998585 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:49.998833 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:49.998837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:49.999218 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:49.999443 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:49.999448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.001625 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.001626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.002420 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.002423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.002762 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.002763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.002838 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.003195 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:50.003199 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:50.004737 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:50.007269 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:50.007270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:50.007270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:50.007459 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:50.007668 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:50.007668 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:50.007668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:50.007720 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:50.007988 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:50.007989 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:50.008038 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 13:17:50.008375 (+ 337us) client_negotiation.cc:240] Negotiation successful
0925 13:17:50.008472 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260925 13:17:50.010834 7600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34171
I20260925 13:17:50.010829 7601 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44189
I20260925 13:17:50.010840 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44113
I20260925 13:17:50.011031 7601 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:50.011045 7600 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:50.011044 7602 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:50.011266 7601 heartbeater.cc:511] Master 127.0.98.60:44189 requested a full tablet report, sending...
I20260925 13:17:50.011267 7600 heartbeater.cc:511] Master 127.0.98.62:34171 requested a full tablet report, sending...
I20260925 13:17:50.011267 7602 heartbeater.cc:511] Master 127.0.98.61:44113 requested a full tablet report, sending...
I20260925 13:17:50.011770 6500 ts_manager.cc:194] Re-registered known tserver with Master: 62cf9c8eb1f74bdaa5e251667e493263 (127.0.98.2:45815)
I20260925 13:17:50.011794 7284 ts_manager.cc:194] Registered new tserver with Master: 62cf9c8eb1f74bdaa5e251667e493263 (127.0.98.2:45815)
I20260925 13:17:50.011809 7097 ts_manager.cc:194] Registered new tserver with Master: 62cf9c8eb1f74bdaa5e251667e493263 (127.0.98.2:45815)
I20260925 13:17:50.012203 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38645
I20260925 13:17:50.012375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39831
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34171
--webserver_port=42107
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34171,127.0.98.61:44113
--tserver_master_addrs=127.0.98.62:34171,127.0.98.61:44113,127.0.98.60:44189 with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:50.094254 7607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:50.094475 7607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:50.094501 7607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:50.095726 7607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:50.095765 7607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:50.095794 7607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:17:50.097338 7607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42107
--tserver_master_addrs=127.0.98.62:34171,127.0.98.61:44113,127.0.98.60:44189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:50.097617 7607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:50.097854 7607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:50.102699 7612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:50.102708 7613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:50.102717 7607 server_base.cc:1034] running on GCE node
W20260925 13:17:50.102676 7615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:50.103051 7607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:50.103304 7607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:50.104460 7607 hybrid_clock.cc:648] HybridClock initialized: now 1790342270104444 us; error 33 us; skew 500 ppm
Sep 25 13:17:50 dist-test-slave-t8d9 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342270, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:50.106968 7607 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 13:17:50.107405 7607 webserver.cc:460] Webserver started at http://127.0.98.3:42107/ using document root <none> and password file <none>
I20260925 13:17:50.107631 7607 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:50.107693 7607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:50.109165 7607 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:50.109920 7622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:50.110080 7607 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:50.110178 7607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "a0f8a13eb91545bfa682c88cef4201a2"
format_stamp: "Formatted at 2026-09-25 13:17:49 on dist-test-slave-t8d9"
server_key: "65316004c03189bbe7a47b47be24bfee"
server_key_iv: "d0a8c22ee77e47024deda5a93db115db"
server_key_version: "encryptionkey@0"
I20260925 13:17:50.110558 7607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:50.130162 7607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:50.132184 7607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:50.132318 7607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:50.132591 7607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:50.132921 7607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:50.132954 7607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:50.132977 7607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:50.132992 7607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:50.138618 7607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39831
I20260925 13:17:50.138686 7735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39831 every 8 connection(s)
I20260925 13:17:50.139034 7607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:17:50 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342270, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 13:17:50 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342270, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 13:17:50 dist-test-slave-t8d9 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342270, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 13:17:50.147369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 7607
I20260925 13:17:50.150013 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.140253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44321 (local address 127.0.98.61:44113)
0925 13:17:50.140293 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.140295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.140319 (+ 24us) server_negotiation.cc:407] Connection header received
0925 13:17:50.140658 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.140660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.140671 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.140700 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:50.141477 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.142014 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.143123 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.143313 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.143376 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.146441 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:50.146443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:50.146443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:50.146451 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:50.148500 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.149032 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.149033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.149034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.149083 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.149263 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.149264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.149264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.149394 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:50.149431 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.149826 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.149913 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:17:50.150012 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.139770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52729 (local address 127.0.98.60:44189)
0925 13:17:50.139815 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.139818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.140176 (+ 358us) server_negotiation.cc:407] Connection header received
0925 13:17:50.140645 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.140646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.140658 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.140708 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:50.142650 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.143406 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.144298 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.144589 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.144657 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.146994 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:50.146996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:50.146996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:50.147004 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:50.148483 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.148893 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.148894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.148895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.148951 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.149216 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.149216 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.149217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.149348 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:50.149391 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.149826 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.149913 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:17:50.150038 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.139787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36343 (local address 127.0.98.62:34171)
0925 13:17:50.139833 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.139835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.140024 (+ 189us) server_negotiation.cc:407] Connection header received
0925 13:17:50.140763 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.140764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.140774 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.140816 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:50.142097 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.142561 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.143563 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.143767 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.143829 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.146763 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:50.146765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:50.146765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:50.146774 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:50.148308 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.148842 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.148843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.148844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.148908 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.149228 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.149229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.149229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.149364 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:50.149407 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.149832 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.149928 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:17:50.150651 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.140175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44113 (local address 127.0.98.3:44321)
0925 13:17:50.140268 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:50.140276 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:50.140559 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:50.141152 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:50.141153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:50.141276 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:50.141384 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.141390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.142124 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.142124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.143018 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.143020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.143382 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.143384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.143530 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.144111 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:50.144122 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:50.146343 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:50.148765 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:50.148766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:50.148767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:50.148954 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:50.149149 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:50.149150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:50.149150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:50.149199 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:50.149473 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:50.149473 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:50.149669 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 13:17:50.150161 (+ 492us) client_negotiation.cc:240] Negotiation successful
0925 13:17:50.150489 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":198,"thread_start_us":58,"threads_started":1}
I20260925 13:17:50.150651 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.139746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.98.3:52729)
0925 13:17:50.140114 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:50.140124 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:50.140554 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:50.140810 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:50.140815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:50.141163 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:50.141396 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.141398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.143479 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.143480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.144214 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.144216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.144668 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.144669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.144731 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.145073 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:50.145077 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:50.146873 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:50.148578 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:50.148579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:50.148580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:50.148788 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:50.149035 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:50.149036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:50.149037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:50.149148 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:50.149437 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:50.149454 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:50.149669 (+ 215us) client_negotiation.cc:769] Sending connection context
0925 13:17:50.150145 (+ 476us) client_negotiation.cc:240] Negotiation successful
0925 13:17:50.150374 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":309,"thread_start_us":67,"threads_started":1}
I20260925 13:17:50.150651 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.139747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:36343)
0925 13:17:50.139906 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:50.139944 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:50.140549 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:50.140836 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:50.140837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:50.141161 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:50.141383 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.141387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.142632 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.142633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.143490 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.143492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.143838 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.143838 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.143908 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.144233 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:50.144237 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:50.146681 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:50.148382 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:50.148391 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:50.148394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:50.148748 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:50.148981 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:50.148981 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:50.148982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:50.149147 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:50.149454 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:50.149455 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:50.149669 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 13:17:50.150304 (+ 635us) client_negotiation.cc:240] Negotiation successful
0925 13:17:50.150433 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260925 13:17:50.151185 7738 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44113
I20260925 13:17:50.151190 7737 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44189
I20260925 13:17:50.151299 7737 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:50.151305 7736 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34171
I20260925 13:17:50.151299 7738 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:50.151389 7736 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:50.151562 7737 heartbeater.cc:511] Master 127.0.98.60:44189 requested a full tablet report, sending...
I20260925 13:17:50.151562 7736 heartbeater.cc:511] Master 127.0.98.62:34171 requested a full tablet report, sending...
I20260925 13:17:50.151562 7738 heartbeater.cc:511] Master 127.0.98.61:44113 requested a full tablet report, sending...
I20260925 13:17:50.151952 6500 ts_manager.cc:194] Re-registered known tserver with Master: a0f8a13eb91545bfa682c88cef4201a2 (127.0.98.3:39831)
I20260925 13:17:50.152009 7097 ts_manager.cc:194] Registered new tserver with Master: a0f8a13eb91545bfa682c88cef4201a2 (127.0.98.3:39831)
I20260925 13:17:50.152410 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36343
I20260925 13:17:50.152611 7284 ts_manager.cc:194] Registered new tserver with Master: a0f8a13eb91545bfa682c88cef4201a2 (127.0.98.3:39831)
I20260925 13:17:50.158653 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.148635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.0.98.62:34171)
0925 13:17:50.150148 (+ 1513us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.150150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.150158 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:17:50.150182 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.150183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.150196 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.150267 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:50.152201 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.152949 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.154004 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.154299 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.154354 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.155893 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:50.155895 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:50.155895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:50.155902 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:50.157456 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.157867 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.157868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.157869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.157930 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.158133 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.158134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.158134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.158268 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:50.158360 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.158475 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.158542 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1494,"thread_start_us":92,"threads_started":1}
I20260925 13:17:50.159499 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.149348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34984 (local address 127.0.98.60:44189)
0925 13:17:50.149615 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.149617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.152352 (+ 2735us) server_negotiation.cc:407] Connection header received
0925 13:17:50.152372 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.152372 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.152385 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.152434 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:50.153078 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.153802 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.154857 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.155148 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.155198 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.156405 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:50.156406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:50.156407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:50.156413 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:50.158187 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.158520 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.158521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.158521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.158577 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.159023 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.159024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.159024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.159155 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:50.159236 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.159325 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.159403 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":53,"threads_started":1}
I20260925 13:17:50.160895 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.148997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.0.98.61:44113)
0925 13:17:50.149806 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.149808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.152208 (+ 2400us) server_negotiation.cc:407] Connection header received
0925 13:17:50.154277 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.154278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.154293 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.154355 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:50.155061 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.155794 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.156690 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.156974 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.157037 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.158172 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:50.158173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:50.158174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:50.158182 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:50.159984 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.160282 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.160283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.160283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.160343 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.160477 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.160478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.160478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.160595 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:50.160678 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.160734 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.160807 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":52,"threads_started":1}
I20260925 13:17:50.160337 6500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:17:50.162237 6500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:50.169694 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.165816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45815 (local address 127.0.0.1:52446)
0925 13:17:50.165863 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:50.165873 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:50.165909 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:50.166521 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:50.166522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:50.166524 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:50.166557 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.166558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.167330 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.167331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.168473 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.168475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.169348 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.169349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.169409 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.169413 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:50.169456 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:17:50.169496 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":27}
I20260925 13:17:50.169705 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.165905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35095 (local address 127.0.0.1:39276)
0925 13:17:50.166056 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:50.166065 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:50.166104 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:50.166404 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:50.166405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:50.166407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:50.166462 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.166464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.167362 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.167364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.168282 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.168285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.169431 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.169432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.169499 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.169505 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 13:17:50.169564 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 13:17:50.169607 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":23,"thread_start_us":79,"threads_started":1}
I20260925 13:17:50.170483 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.165909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.0.98.2:45815)
0925 13:17:50.166383 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.166388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.166397 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:50.166432 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.166434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.166450 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.166525 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:50.166592 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.167251 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.168555 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.169261 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.169339 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.170075 (+ 736us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.170121 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.170177 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":70,"threads_started":1}
I20260925 13:17:50.170593 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.166098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.0.98.1:35095)
0925 13:17:50.166250 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.166255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.166265 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:17:50.166303 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.166306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.166322 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.166409 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:50.166612 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.167277 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.168345 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.169344 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.169748 (+ 404us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.169856 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.169884 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.169925 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260925 13:17:50.170676 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.165927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.0.98.3:39831)
0925 13:17:50.167396 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.167400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.167408 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:17:50.167444 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.167446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.167462 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.167535 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:50.167643 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.168698 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.169630 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.170341 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.170396 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.170464 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.170562 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.170601 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1431,"thread_start_us":87,"threads_started":1}
I20260925 13:17:50.170774 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.165816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39831 (local address 127.0.0.1:49176)
0925 13:17:50.166176 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:50.166183 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:50.166216 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:50.167541 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:50.167542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:50.167545 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:50.167586 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.167587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.168760 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.168761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.169518 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.169519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.170419 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.170421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.170482 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.170486 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:50.170531 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 13:17:50.170712 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":133,"thread_start_us":62,"threads_started":1}
I20260925 13:17:50.171365 7670 tablet_service.cc:1467] Processing CreateTablet for tablet df36094f6672495fb7efcaf42e327b5c (DEFAULT_TABLE table=test-table [id=04b43c082d9a42dcbf255c7529dfc59a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:50.171311 7397 tablet_service.cc:1467] Processing CreateTablet for tablet df36094f6672495fb7efcaf42e327b5c (DEFAULT_TABLE table=test-table [id=04b43c082d9a42dcbf255c7529dfc59a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:50.171685 7670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df36094f6672495fb7efcaf42e327b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:50.171685 7397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df36094f6672495fb7efcaf42e327b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:50.174256 7534 tablet_service.cc:1467] Processing CreateTablet for tablet df36094f6672495fb7efcaf42e327b5c (DEFAULT_TABLE table=test-table [id=04b43c082d9a42dcbf255c7529dfc59a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:50.174552 7534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df36094f6672495fb7efcaf42e327b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:50.175226 7762 tablet_bootstrap.cc:492] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04: Bootstrap starting.
I20260925 13:17:50.175545 7763 tablet_bootstrap.cc:492] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2: Bootstrap starting.
I20260925 13:17:50.176030 7762 tablet_bootstrap.cc:654] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:50.176045 7763 tablet_bootstrap.cc:654] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:50.176302 7763 log.cc:824] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:50.176334 7762 log.cc:824] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:50.177013 7762 tablet_bootstrap.cc:492] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04: No bootstrap required, opened a new log
I20260925 13:17:50.177007 7763 tablet_bootstrap.cc:492] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2: No bootstrap required, opened a new log
I20260925 13:17:50.177070 7763 ts_tablet_manager.cc:1397] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:17:50.177070 7762 ts_tablet_manager.cc:1397] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:17:50.177623 7766 tablet_bootstrap.cc:492] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263: Bootstrap starting.
I20260925 13:17:50.178124 7766 tablet_bootstrap.cc:654] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:50.178316 7766 log.cc:824] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:50.178871 7766 tablet_bootstrap.cc:492] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263: No bootstrap required, opened a new log
I20260925 13:17:50.178913 7766 ts_tablet_manager.cc:1397] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:50.179092 7762 raft_consensus.cc:348] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } } peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } }
I20260925 13:17:50.179250 7762 raft_consensus.cc:374] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:50.179279 7762 raft_consensus.cc:729] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fb1305a6a8a403cb15b4b556c796b04, State: Initialized, Role: FOLLOWER
I20260925 13:17:50.179428 7762 consensus_queue.cc:260] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } } peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } }
I20260925 13:17:50.179425 7763 raft_consensus.cc:348] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } } peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } }
I20260925 13:17:50.179562 7763 raft_consensus.cc:374] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:50.179594 7763 raft_consensus.cc:729] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f8a13eb91545bfa682c88cef4201a2, State: Initialized, Role: FOLLOWER
I20260925 13:17:50.179721 7762 ts_tablet_manager.cc:1428] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:17:50.179697 7763 consensus_queue.cc:260] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } } peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } }
I20260925 13:17:50.179739 7463 heartbeater.cc:503] Master 127.0.98.62:34171 was elected leader, sending a full tablet report...
I20260925 13:17:50.179935 7763 ts_tablet_manager.cc:1428] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:17:50.180321 7736 heartbeater.cc:503] Master 127.0.98.62:34171 was elected leader, sending a full tablet report...
I20260925 13:17:50.180393 7766 raft_consensus.cc:348] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } } peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } }
I20260925 13:17:50.180541 7766 raft_consensus.cc:374] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:50.180598 7766 raft_consensus.cc:729] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62cf9c8eb1f74bdaa5e251667e493263, State: Initialized, Role: FOLLOWER
I20260925 13:17:50.180701 7766 consensus_queue.cc:260] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } } peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } }
I20260925 13:17:50.181059 7766 ts_tablet_manager.cc:1428] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:17:50.181171 7600 heartbeater.cc:503] Master 127.0.98.62:34171 was elected leader, sending a full tablet report...
I20260925 13:17:50.207073 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.204409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:34990)
0925 13:17:50.204470 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:50.204484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:50.204537 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:50.204726 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:50.204726 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:50.204728 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:50.204775 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.204776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.205465 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.205466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.206280 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.206284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.206876 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.206877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.206927 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.206931 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:50.206959 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 13:17:50.207014 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 13:17:50.207072 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.204516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.0.98.60:44189)
0925 13:17:50.204575 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.204577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.204588 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:17:50.204618 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.204619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.204643 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.204711 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:50.204862 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.205372 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.206384 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.206808 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.206873 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.206962 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.206977 (+ 15us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.207012 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:17:50.207581 7294 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260925 13:17:50.208140 7770 raft_consensus.cc:482] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:50.208249 7770 raft_consensus.cc:504] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } } peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } }
I20260925 13:17:50.208534 7326 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ced1fac976141779f951abf99ce6599. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } } }
I20260925 13:17:50.208624 7326 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [sys.catalog]: This master's current role is: LEARNER
I20260925 13:17:50.208636 7770 leader_election.cc:288] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0f8a13eb91545bfa682c88cef4201a2 (127.0.98.3:39831), 0fb1305a6a8a403cb15b4b556c796b04 (127.0.98.1:35095)
I20260925 13:17:50.211795 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.208958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46323 (local address 127.0.98.1:35095)
0925 13:17:50.209001 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.209003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.209010 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:17:50.209033 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.209034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.209044 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.209100 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:50.209360 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.209893 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.211081 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.211458 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.211525 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.211536 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.211682 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.211726 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:17:50.211812 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.208785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35095 (local address 127.0.98.2:46323)
0925 13:17:50.208827 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:50.208839 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:50.208877 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:50.209122 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:50.209123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:50.209125 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:50.209212 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.209214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.209973 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.209974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.210988 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.210991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.211535 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.211535 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.211607 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.211610 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:50.211659 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 13:17:50.211699 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 13:17:50.211825 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.208786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39831 (local address 127.0.98.2:36923)
0925 13:17:50.209179 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:50.209188 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:50.209225 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:50.209391 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:50.209392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:50.209393 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:50.209454 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.209456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.210083 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.210084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.210996 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:50.210998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.211539 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.211540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:50.211636 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.211640 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:50.211686 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 13:17:50.211725 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":350}
I20260925 13:17:50.211971 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.208901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36923 (local address 127.0.98.3:39831)
0925 13:17:50.208943 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.208945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.209274 (+ 329us) server_negotiation.cc:407] Connection header received
0925 13:17:50.209323 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.209324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.209336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.209376 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:50.209518 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.209998 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.211084 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.211468 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.211527 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.211537 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.211852 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.211898 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:17:50.212368 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df36094f6672495fb7efcaf42e327b5c" candidate_uuid: "62cf9c8eb1f74bdaa5e251667e493263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fb1305a6a8a403cb15b4b556c796b04" is_pre_election: true
I20260925 13:17:50.212430 7690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df36094f6672495fb7efcaf42e327b5c" candidate_uuid: "62cf9c8eb1f74bdaa5e251667e493263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0f8a13eb91545bfa682c88cef4201a2" is_pre_election: true
I20260925 13:17:50.212507 7417 raft_consensus.cc:2454] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62cf9c8eb1f74bdaa5e251667e493263 in term 0.
I20260925 13:17:50.212561 7690 raft_consensus.cc:2454] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62cf9c8eb1f74bdaa5e251667e493263 in term 0.
I20260925 13:17:50.212774 7489 leader_election.cc:302] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62cf9c8eb1f74bdaa5e251667e493263, a0f8a13eb91545bfa682c88cef4201a2; no voters:
I20260925 13:17:50.212939 7770 raft_consensus.cc:2781] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:50.213009 7770 raft_consensus.cc:482] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:50.213035 7770 raft_consensus.cc:3037] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:50.213778 7770 raft_consensus.cc:504] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } } peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } }
I20260925 13:17:50.213914 7770 leader_election.cc:288] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [CANDIDATE]: Term 1 election: Requested vote from peers a0f8a13eb91545bfa682c88cef4201a2 (127.0.98.3:39831), 0fb1305a6a8a403cb15b4b556c796b04 (127.0.98.1:35095)
I20260925 13:17:50.214078 7690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df36094f6672495fb7efcaf42e327b5c" candidate_uuid: "62cf9c8eb1f74bdaa5e251667e493263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0f8a13eb91545bfa682c88cef4201a2"
I20260925 13:17:50.214082 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df36094f6672495fb7efcaf42e327b5c" candidate_uuid: "62cf9c8eb1f74bdaa5e251667e493263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fb1305a6a8a403cb15b4b556c796b04"
I20260925 13:17:50.214160 7690 raft_consensus.cc:3037] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:50.214162 7417 raft_consensus.cc:3037] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:50.215030 7690 raft_consensus.cc:2454] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62cf9c8eb1f74bdaa5e251667e493263 in term 1.
I20260925 13:17:50.215031 7417 raft_consensus.cc:2454] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62cf9c8eb1f74bdaa5e251667e493263 in term 1.
I20260925 13:17:50.215191 7489 leader_election.cc:302] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62cf9c8eb1f74bdaa5e251667e493263, a0f8a13eb91545bfa682c88cef4201a2; no voters:
I20260925 13:17:50.215265 7770 raft_consensus.cc:2781] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:50.215421 7770 raft_consensus.cc:686] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [term 1 LEADER]: Becoming Leader. State: Replica: 62cf9c8eb1f74bdaa5e251667e493263, State: Running, Role: LEADER
I20260925 13:17:50.215508 7770 consensus_queue.cc:237] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } } peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } }
I20260925 13:17:50.216411 6499 catalog_manager.cc:5495] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62cf9c8eb1f74bdaa5e251667e493263 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "62cf9c8eb1f74bdaa5e251667e493263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } health_report { overall_health: UNKNOWN } } }
I20260925 13:17:50.217033 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: true } } }
I20260925 13:17:50.217141 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:50.218103 7777 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599: attempting to promote NON_VOTER ee3b3de7a7ca49a4b136c62850c91065 to VOTER
I20260925 13:17:50.218225 7777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } }
I20260925 13:17:50.218603 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 FOLLOWER]: Refusing update from remote peer 0ced1fac976141779f951abf99ce6599: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 13:17:50.218712 7294 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 1 LEARNER]: Refusing update from remote peer 0ced1fac976141779f951abf99ce6599: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 13:17:50.218770 7248 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 13:17:50.218978 7248 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 13:17:50.219365 7777 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ee3b3de7a7ca49a4b136c62850c91065 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } } }
I20260925 13:17:50.219475 7107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ee3b3de7a7ca49a4b136c62850c91065 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } } }
I20260925 13:17:50.219624 7294 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ee3b3de7a7ca49a4b136c62850c91065 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } } }
I20260925 13:17:50.220309 7777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } } }
I20260925 13:17:50.220386 7777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:50.220350 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } } }
I20260925 13:17:50.220418 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:17:50.220438 7777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } } }
I20260925 13:17:50.220391 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } } }
I20260925 13:17:50.220547 7326 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ced1fac976141779f951abf99ce6599" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } } }
I20260925 13:17:50.220625 7326 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:17:50.220515 7777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:50.220525 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:50.227164 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.224673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.0.98.2:45815)
0925 13:17:50.224714 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.224715 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.224724 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:50.224803 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.224803 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.224823 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.224880 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:50.224993 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.225448 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.226259 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.226470 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.226535 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.226657 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:50.226767 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:50.226831 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.227057 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.227087 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:17:50.229139 7417 raft_consensus.cc:1261] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04 [term 1 FOLLOWER]: Refusing update from remote peer 62cf9c8eb1f74bdaa5e251667e493263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:50.229291 7690 raft_consensus.cc:1261] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2 [term 1 FOLLOWER]: Refusing update from remote peer 62cf9c8eb1f74bdaa5e251667e493263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:50.229589 7770 consensus_queue.cc:1035] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:50.229802 7775 consensus_queue.cc:1035] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:50.233207 7782 mvcc.cc:205] Tried to move back new op lower bound from 7333241938855505920 to 7333241938803392512. Current Snapshot: MvccSnapshot[applied={T|T < 7333241938855505920}]
I20260925 13:17:50.233474 7780 mvcc.cc:205] Tried to move back new op lower bound from 7333241938855505920 to 7333241938803392512. Current Snapshot: MvccSnapshot[applied={T|T < 7333241938855505920}]
I20260925 13:17:50.235637 6498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:50.235720 6498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:50.236974 6498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ced1fac976141779f951abf99ce6599: Sending DeleteTablet for 3 replicas of tablet df36094f6672495fb7efcaf42e327b5c
I20260925 13:17:50.237355 7397 tablet_service.cc:1514] Processing DeleteTablet for tablet df36094f6672495fb7efcaf42e327b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39276
I20260925 13:17:50.237645 7670 tablet_service.cc:1514] Processing DeleteTablet for tablet df36094f6672495fb7efcaf42e327b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49176
I20260925 13:17:50.237692 7534 tablet_service.cc:1514] Processing DeleteTablet for tablet df36094f6672495fb7efcaf42e327b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52446
I20260925 13:17:50.237860 7789 tablet_replica.cc:330] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2: stopping tablet replica
I20260925 13:17:50.237964 7789 raft_consensus.cc:2229] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:50.238089 7789 raft_consensus.cc:2258] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:50.238108 7788 tablet_replica.cc:330] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04: stopping tablet replica
I20260925 13:17:50.238281 7788 raft_consensus.cc:2229] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:50.238464 7788 raft_consensus.cc:2258] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:50.238624 7789 ts_tablet_manager.cc:1905] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:50.238790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 6468
I20260925 13:17:50.241175 7789 ts_tablet_manager.cc:1918] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:50.241246 7789 log.cc:1196] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/df36094f6672495fb7efcaf42e327b5c
I20260925 13:17:50.241478 7789 ts_tablet_manager.cc:1939] T df36094f6672495fb7efcaf42e327b5c P a0f8a13eb91545bfa682c88cef4201a2: Deleting consensus metadata
W20260925 13:17:50.244870 7625 connection.cc:535] server connection from 127.0.0.1:49176 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 13:17:50.245231 7790 tablet_replica.cc:330] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263: stopping tablet replica
I20260925 13:17:50.245358 7790 raft_consensus.cc:2229] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:17:50.245514 7790 raft_consensus.cc:2258] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:50.246004 7788 ts_tablet_manager.cc:1905] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:50.246002 7790 ts_tablet_manager.cc:1905] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:50.247682 7788 ts_tablet_manager.cc:1918] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:50.247766 7788 log.cc:1196] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-0/wal/wals/df36094f6672495fb7efcaf42e327b5c
I20260925 13:17:50.248106 7788 ts_tablet_manager.cc:1939] T df36094f6672495fb7efcaf42e327b5c P 0fb1305a6a8a403cb15b4b556c796b04: Deleting consensus metadata
W20260925 13:17:50.248517 7350 connection.cc:441] server connection from 127.0.0.1:39276 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39276 (request call id 1) could send its response
I20260925 13:17:50.249650 7790 ts_tablet_manager.cc:1918] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:50.249732 7790 log.cc:1196] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/df36094f6672495fb7efcaf42e327b5c
I20260925 13:17:50.249984 7790 ts_tablet_manager.cc:1939] T df36094f6672495fb7efcaf42e327b5c P 62cf9c8eb1f74bdaa5e251667e493263: Deleting consensus metadata
W20260925 13:17:50.250540 7490 connection.cc:441] server connection from 127.0.0.1:52446 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52446 (request call id 1) could send its response
I20260925 13:17:50.252748 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.237966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.0.98.60:44189)
0925 13:17:50.238004 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.238007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.245103 (+ 7096us) server_negotiation.cc:407] Connection header received
0925 13:17:50.245156 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.245157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.245169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.245216 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:50.245944 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.246624 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.247427 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.247721 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.247783 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.249638 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:50.249640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:50.249640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:50.249647 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:50.251209 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.251559 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.251560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.251561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.251630 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.251794 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.251794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.251795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.251903 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:50.251986 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.252578 (+ 592us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.252657 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:17:50.255924 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.247849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.0.98.61:44113)
0925 13:17:50.247893 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.247895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.248816 (+ 921us) server_negotiation.cc:407] Connection header received
0925 13:17:50.248909 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.248910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.248926 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.248956 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:50.249662 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.250360 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.251652 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.251980 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.252019 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.253596 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:50.253597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:50.253598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:50.253605 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:50.254922 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.255233 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.255234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.255234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.255287 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.255447 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.255447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.255448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.255557 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:50.255648 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.255746 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.255830 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:17:50.256451 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:50.249464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.0.98.60:44189)
0925 13:17:50.249507 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:50.249509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:50.249517 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:17:50.249556 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:50.249557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:50.249566 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:50.249613 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:50.250287 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.250962 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.251943 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:50.252220 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:50.252256 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:50.253640 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:50.253642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:50.253643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:50.253649 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:50.255170 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.255537 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.255537 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.255538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.255591 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:50.255772 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:50.255772 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:50.255773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:50.255899 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:50.255974 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:50.256289 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 13:17:50.256363 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:17:51.235054 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.234774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:60129)
0925 13:17:51.234919 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.234998 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260925 13:17:51.235112 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.234775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:47107)
0925 13:17:51.235025 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.235077 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":68,"threads_started":1}
I20260925 13:17:51.235121 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.234897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:49335)
0925 13:17:51.235002 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.235073 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":44,"threads_started":1}
W20260925 13:17:51.235250 7600 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34171 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
W20260925 13:17:51.235250 7463 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34171 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
W20260925 13:17:51.235428 7736 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34171 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
I20260925 13:17:51.737288 7802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0ced1fac976141779f951abf99ce6599)
I20260925 13:17:51.737427 7802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } }
I20260925 13:17:51.737730 7802 leader_election.cc:288] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ced1fac976141779f951abf99ce6599 (127.0.98.62:34171), ee3b3de7a7ca49a4b136c62850c91065 (127.0.98.60:44189)
I20260925 13:17:51.738089 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.737848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.0.1:52510)
0925 13:17:51.737979 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.738044 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260925 13:17:51.738087 7804 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0ced1fac976141779f951abf99ce6599)
W20260925 13:17:51.738235 7082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
I20260925 13:17:51.738209 7804 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } }
I20260925 13:17:51.738443 7804 leader_election.cc:288] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ced1fac976141779f951abf99ce6599 (127.0.98.62:34171), 9318f44ac5814b20b91da5f65f9d1f20 (127.0.98.61:44113)
I20260925 13:17:51.738703 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.738522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.0.1:52518)
0925 13:17:51.738618 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.738662 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":46,"threads_started":1}
I20260925 13:17:51.738782 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.738699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.0.1:52530)
0925 13:17:51.738737 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.738758 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 13:17:51.738859 7082 leader_election.cc:334] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0ced1fac976141779f951abf99ce6599 (127.0.98.62:34171): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
W20260925 13:17:51.738922 7269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
I20260925 13:17:51.739341 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.739222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.0.1:52540)
0925 13:17:51.739268 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.739305 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 13:17:51.739414 7269 leader_election.cc:334] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0ced1fac976141779f951abf99ce6599 (127.0.98.62:34171): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
I20260925 13:17:51.741246 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.738047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35016 (local address 127.0.98.60:44189)
0925 13:17:51.738140 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:51.738142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:51.738151 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:51.738413 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:51.738414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:51.738426 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:51.739098 (+ 672us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:51.739132 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.739681 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.740604 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.740962 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.741018 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:51.741028 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:51.741151 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 13:17:51.741184 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":48,"threads_started":1}
I20260925 13:17:51.741259 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.737875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:35016)
0925 13:17:51.737995 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.738006 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:51.738060 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:51.738978 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:51.738979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:51.738981 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:51.739037 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:51.739040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.739749 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.739750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:51.740525 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:51.740527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.741026 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.741027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:51.741091 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:51.741095 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:51.741142 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 13:17:51.741184 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":48,"threads_started":1}
I20260925 13:17:51.741423 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.738567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44113 (local address 127.0.0.1:60072)
0925 13:17:51.738760 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.738768 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:51.738826 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:51.739014 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:51.739016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:51.739019 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:51.739068 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:51.739071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.739954 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.739955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:51.740686 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:51.740687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.741193 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.741194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:51.741259 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:51.741262 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:51.741312 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 13:17:51.741352 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":46,"threads_started":1}
I20260925 13:17:51.741472 7294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9318f44ac5814b20b91da5f65f9d1f20" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ee3b3de7a7ca49a4b136c62850c91065" is_pre_election: true
I20260925 13:17:51.741603 7294 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9318f44ac5814b20b91da5f65f9d1f20 in term 1.
I20260925 13:17:51.741761 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.738611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60072 (local address 127.0.98.61:44113)
0925 13:17:51.738736 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:51.738738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:51.738814 (+ 76us) server_negotiation.cc:407] Connection header received
0925 13:17:51.738882 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:51.738883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:51.738893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:51.738963 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:51.739153 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.739878 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.740764 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.741128 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.741185 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:51.741194 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:51.741647 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 13:17:51.741691 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260925 13:17:51.741775 7083 leader_election.cc:302] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9318f44ac5814b20b91da5f65f9d1f20, ee3b3de7a7ca49a4b136c62850c91065; no voters: 0ced1fac976141779f951abf99ce6599
I20260925 13:17:51.741905 7802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 13:17:51.741947 7802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0ced1fac976141779f951abf99ce6599)
I20260925 13:17:51.741973 7802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 1 FOLLOWER]: Advancing to term 2
I20260925 13:17:51.741993 7107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee3b3de7a7ca49a4b136c62850c91065" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9318f44ac5814b20b91da5f65f9d1f20" is_pre_election: true
I20260925 13:17:51.742671 7802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } }
I20260925 13:17:51.742807 7107 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee3b3de7a7ca49a4b136c62850c91065 in current term 2: Already voted for candidate 9318f44ac5814b20b91da5f65f9d1f20 in this term.
I20260925 13:17:51.742816 7802 leader_election.cc:288] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [CANDIDATE]: Term 2 election: Requested vote from peers 0ced1fac976141779f951abf99ce6599 (127.0.98.62:34171), ee3b3de7a7ca49a4b136c62850c91065 (127.0.98.60:44189)
I20260925 13:17:51.742954 7294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9318f44ac5814b20b91da5f65f9d1f20" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ee3b3de7a7ca49a4b136c62850c91065"
I20260925 13:17:51.742980 7270 leader_election.cc:302] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee3b3de7a7ca49a4b136c62850c91065; no voters: 0ced1fac976141779f951abf99ce6599, 9318f44ac5814b20b91da5f65f9d1f20
I20260925 13:17:51.743021 7294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 1 FOLLOWER]: Advancing to term 2
I20260925 13:17:51.743180 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.742874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.0.1:52556)
0925 13:17:51.743127 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.743156 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260925 13:17:51.743441 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.743362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.0.1:52560)
0925 13:17:51.743400 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.743422 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 13:17:51.743498 7082 leader_election.cc:334] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0ced1fac976141779f951abf99ce6599 (127.0.98.62:34171): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
I20260925 13:17:51.743733 7294 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9318f44ac5814b20b91da5f65f9d1f20 in term 2.
I20260925 13:17:51.743826 7804 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 13:17:51.743913 7083 leader_election.cc:302] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9318f44ac5814b20b91da5f65f9d1f20, ee3b3de7a7ca49a4b136c62850c91065; no voters: 0ced1fac976141779f951abf99ce6599
I20260925 13:17:51.743986 7802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 13:17:51.744120 7802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [term 2 LEADER]: Becoming Leader. State: Replica: 9318f44ac5814b20b91da5f65f9d1f20, State: Running, Role: LEADER
I20260925 13:17:51.744179 7802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } }
I20260925 13:17:51.744451 7812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9318f44ac5814b20b91da5f65f9d1f20. Latest consensus state: current_term: 2 leader_uuid: "9318f44ac5814b20b91da5f65f9d1f20" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } } }
I20260925 13:17:51.744539 7812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:51.744706 7814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:51.745100 7814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:51.745167 7814 catalog_manager.cc:1259] Loaded cluster ID: a126f6acf3074ab08732f62e7f3ce19a
I20260925 13:17:51.745189 7814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:17:51.745345 7814 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:51.745446 7814 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: Loaded TSK: 0
I20260925 13:17:51.745626 7814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:17:51.789315 7096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:17:51.790920 7096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:51.791875 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.791772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.0.1:52574)
0925 13:17:51.791818 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.791848 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 13:17:51.791894 7294 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [term 2 FOLLOWER]: Refusing update from remote peer 9318f44ac5814b20b91da5f65f9d1f20: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 13:17:51.792188 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.792093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.0.1:52590)
0925 13:17:51.792137 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.792165 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 13:17:51.792186 7812 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 13:17:51.792304 7082 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 -> Peer 0ced1fac976141779f951abf99ce6599 (127.0.98.62:34171): Couldn't send request to peer 0ced1fac976141779f951abf99ce6599. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 13:17:51.792769 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9318f44ac5814b20b91da5f65f9d1f20. Latest consensus state: current_term: 2 leader_uuid: "9318f44ac5814b20b91da5f65f9d1f20" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } } }
I20260925 13:17:51.792855 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:17:51.793236 7812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9318f44ac5814b20b91da5f65f9d1f20" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } } }
I20260925 13:17:51.793320 7812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:51.793442 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9318f44ac5814b20b91da5f65f9d1f20" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ced1fac976141779f951abf99ce6599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } peers { permanent_uuid: "9318f44ac5814b20b91da5f65f9d1f20" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44113 } attrs { promote: false } } peers { permanent_uuid: "ee3b3de7a7ca49a4b136c62850c91065" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attrs { promote: false } } }
I20260925 13:17:51.793525 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3b3de7a7ca49a4b136c62850c91065 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:17:51.793820 7816 mvcc.cc:205] Tried to move back new op lower bound from 7333241945257824256 to 7333241945064648704. Current Snapshot: MvccSnapshot[applied={T|T < 7333241945257824256 or (T in {7333241945257824256})}]
I20260925 13:17:51.797974 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.795061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35095 (local address 127.0.0.1:39288)
0925 13:17:51.795091 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.795101 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:51.795145 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:51.795403 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:51.795404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:51.795405 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:51.795446 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:51.795448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.796169 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.796170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:51.797021 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:51.797023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.797781 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.797782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:51.797843 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:51.797848 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:17:51.797873 (+ 25us) client_negotiation.cc:240] Negotiation successful
0925 13:17:51.797909 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260925 13:17:51.798213 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.795211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.0.98.1:35095)
0925 13:17:51.795292 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:51.795294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:51.795302 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:17:51.795328 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:51.795329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:51.795339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:51.795401 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:51.795494 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.796103 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.797096 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.797694 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.798079 (+ 385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:51.798093 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:51.798110 (+ 17us) server_negotiation.cc:299] Negotiation successful
0925 13:17:51.798150 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"thread_start_us":41,"threads_started":1}
I20260925 13:17:51.798288 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.795135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.0.98.2:45815)
0925 13:17:51.795240 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:51.795242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:51.795645 (+ 403us) server_negotiation.cc:407] Connection header received
0925 13:17:51.795674 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:51.795675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:51.795685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:51.795743 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:51.795865 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.796436 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.797515 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.797916 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.797974 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:51.797986 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:51.798177 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 13:17:51.798224 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
I20260925 13:17:51.798446 7397 tablet_service.cc:1467] Processing CreateTablet for tablet 108151639df24d4dbce46edfa7a9d4d0 (DEFAULT_TABLE table=test-table [id=7a3e5caace734837852deb7b46e62d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:51.798504 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.795064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45815 (local address 127.0.0.1:52462)
0925 13:17:51.795590 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.795596 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:51.795635 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:51.795759 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:51.795760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:51.795762 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:51.795797 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:51.795799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.796506 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.796507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:51.797447 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:51.797449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.797983 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.797984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:51.798109 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:51.798113 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:51.798389 (+ 276us) client_negotiation.cc:240] Negotiation successful
0925 13:17:51.798430 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":422,"thread_start_us":63,"threads_started":1}
I20260925 13:17:51.798591 7397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108151639df24d4dbce46edfa7a9d4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:51.798707 7534 tablet_service.cc:1467] Processing CreateTablet for tablet 108151639df24d4dbce46edfa7a9d4d0 (DEFAULT_TABLE table=test-table [id=7a3e5caace734837852deb7b46e62d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:51.798839 7534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108151639df24d4dbce46edfa7a9d4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:51.799412 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.795153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39831 (local address 127.0.0.1:49178)
0925 13:17:51.795285 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.795294 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:51.795338 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:51.796292 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:51.796293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:51.796296 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:51.796338 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:51.796340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.797164 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.797164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:51.798198 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:51.798200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.799192 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.799194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:51.799251 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:51.799254 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:51.799314 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 13:17:51.799356 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 13:17:51.799434 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.795415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.0.98.3:39831)
0925 13:17:51.796194 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:51.796195 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:51.796200 (+ 5us) server_negotiation.cc:407] Connection header received
0925 13:17:51.796219 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:51.796220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:51.796227 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:51.796275 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:51.796397 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.797102 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.798262 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.799110 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.799178 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:51.799191 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:51.799324 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 13:17:51.799361 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":52,"threads_started":1}
I20260925 13:17:51.799921 7825 tablet_bootstrap.cc:492] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04: Bootstrap starting.
I20260925 13:17:51.800312 7825 tablet_bootstrap.cc:654] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:51.800350 7670 tablet_service.cc:1467] Processing CreateTablet for tablet 108151639df24d4dbce46edfa7a9d4d0 (DEFAULT_TABLE table=test-table [id=7a3e5caace734837852deb7b46e62d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:51.800458 7670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108151639df24d4dbce46edfa7a9d4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:51.800616 7826 tablet_bootstrap.cc:492] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263: Bootstrap starting.
I20260925 13:17:51.800836 7825 tablet_bootstrap.cc:492] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04: No bootstrap required, opened a new log
I20260925 13:17:51.800882 7825 ts_tablet_manager.cc:1397] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:51.801031 7825 raft_consensus.cc:348] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } } peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260925 13:17:51.801115 7825 raft_consensus.cc:374] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:51.801144 7825 raft_consensus.cc:729] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fb1305a6a8a403cb15b4b556c796b04, State: Initialized, Role: FOLLOWER
I20260925 13:17:51.801195 7825 consensus_queue.cc:260] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } } peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260925 13:17:51.801347 7825 ts_tablet_manager.cc:1428] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:51.801728 7826 tablet_bootstrap.cc:654] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:51.801869 7830 tablet_bootstrap.cc:492] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2: Bootstrap starting.
I20260925 13:17:51.802037 7465 heartbeater.cc:503] Master 127.0.98.61:44113 was elected leader, sending a full tablet report...
I20260925 13:17:51.802295 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.801480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:47279)
0925 13:17:51.802215 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.802257 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":98,"threads_started":1}
I20260925 13:17:51.802485 7830 tablet_bootstrap.cc:654] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:51.802682 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.802503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:44409)
0925 13:17:51.802630 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.802658 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260925 13:17:51.802744 7826 tablet_bootstrap.cc:492] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263: No bootstrap required, opened a new log
I20260925 13:17:51.802795 7826 ts_tablet_manager.cc:1397] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:17:51.802911 7826 raft_consensus.cc:348] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } } peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260925 13:17:51.802976 7826 raft_consensus.cc:374] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:51.803004 7826 raft_consensus.cc:729] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62cf9c8eb1f74bdaa5e251667e493263, State: Initialized, Role: FOLLOWER
I20260925 13:17:51.803042 7826 consensus_queue.cc:260] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } } peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260925 13:17:51.803088 7830 tablet_bootstrap.cc:492] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2: No bootstrap required, opened a new log
I20260925 13:17:51.803140 7830 ts_tablet_manager.cc:1397] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:51.803174 7826 ts_tablet_manager.cc:1428] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:51.803337 7830 raft_consensus.cc:348] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } } peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260925 13:17:51.803416 7830 raft_consensus.cc:374] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:51.803435 7830 raft_consensus.cc:729] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f8a13eb91545bfa682c88cef4201a2, State: Initialized, Role: FOLLOWER
I20260925 13:17:51.803470 7830 consensus_queue.cc:260] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } } peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260925 13:17:51.803534 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.803347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:51933)
0925 13:17:51.803465 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.803502 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":76,"threads_started":1}
I20260925 13:17:51.803627 7830 ts_tablet_manager.cc:1428] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:51.803880 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.803735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:49887)
0925 13:17:51.803833 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.803855 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":64,"threads_started":1}
I20260925 13:17:51.804030 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.803848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:53947)
0925 13:17:51.803958 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.803990 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":73,"threads_started":1}
I20260925 13:17:51.804184 7602 heartbeater.cc:503] Master 127.0.98.61:44113 was elected leader, sending a full tablet report...
I20260925 13:17:51.804415 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.804319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:47931)
0925 13:17:51.804370 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.804391 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 13:17:51.804512 7738 heartbeater.cc:503] Master 127.0.98.61:44113 was elected leader, sending a full tablet report...
W20260925 13:17:51.859443 7466 tablet.cc:2369] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:17:51.890638 7739 tablet.cc:2369] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:17:51.899051 7828 raft_consensus.cc:482] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:51.899155 7828 raft_consensus.cc:504] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } } peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260925 13:17:51.899472 7828 leader_election.cc:288] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62cf9c8eb1f74bdaa5e251667e493263 (127.0.98.2:45815), a0f8a13eb91545bfa682c88cef4201a2 (127.0.98.3:39831)
I20260925 13:17:51.903227 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.899734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41819 (local address 127.0.98.2:45815)
0925 13:17:51.899791 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:51.899793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:51.899801 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:17:51.899832 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:51.899833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:51.899845 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:51.899922 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:51.900070 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.900827 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.902310 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.902881 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.902942 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:51.902956 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:51.903112 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 13:17:51.903156 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:17:51.903227 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.899688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53517 (local address 127.0.98.3:39831)
0925 13:17:51.899737 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:51.899739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:51.899922 (+ 183us) server_negotiation.cc:407] Connection header received
0925 13:17:51.899966 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:51.899967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:51.899980 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:51.900029 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:51.900119 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.900838 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.902318 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.902876 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.902944 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:51.902956 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:51.903112 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 13:17:51.903158 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:17:51.903257 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.899626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39831 (local address 127.0.98.1:53517)
0925 13:17:51.899860 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.899867 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:51.899912 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:51.900035 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:51.900036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:51.900038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:51.900065 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:51.900066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.901057 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.901058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:51.902215 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:51.902217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.902956 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.902958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:51.903043 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:51.903046 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:51.903097 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 13:17:51.903146 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":116,"thread_start_us":104,"threads_started":1}
I20260925 13:17:51.903261 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.899626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45815 (local address 127.0.98.1:41819)
0925 13:17:51.899688 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.899698 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:51.899750 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:51.899947 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:51.899948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:51.899951 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:51.900009 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:51.900011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.900932 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.900932 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:51.902214 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:51.902217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.902958 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.902959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:51.903042 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:51.903046 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:51.903097 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 13:17:51.903147 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":17}
I20260925 13:17:51.903532 7690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108151639df24d4dbce46edfa7a9d4d0" candidate_uuid: "0fb1305a6a8a403cb15b4b556c796b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0f8a13eb91545bfa682c88cef4201a2" is_pre_election: true
I20260925 13:17:51.903633 7690 raft_consensus.cc:2454] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fb1305a6a8a403cb15b4b556c796b04 in term 0.
I20260925 13:17:51.903676 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108151639df24d4dbce46edfa7a9d4d0" candidate_uuid: "0fb1305a6a8a403cb15b4b556c796b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62cf9c8eb1f74bdaa5e251667e493263" is_pre_election: true
I20260925 13:17:51.903810 7554 raft_consensus.cc:2454] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fb1305a6a8a403cb15b4b556c796b04 in term 0.
I20260925 13:17:51.903800 7352 leader_election.cc:302] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fb1305a6a8a403cb15b4b556c796b04, a0f8a13eb91545bfa682c88cef4201a2; no voters:
I20260925 13:17:51.903927 7828 raft_consensus.cc:2781] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:51.903970 7828 raft_consensus.cc:482] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:51.903987 7828 raft_consensus.cc:3037] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:51.904664 7828 raft_consensus.cc:504] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } } peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260925 13:17:51.904778 7828 leader_election.cc:288] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [CANDIDATE]: Term 1 election: Requested vote from peers 62cf9c8eb1f74bdaa5e251667e493263 (127.0.98.2:45815), a0f8a13eb91545bfa682c88cef4201a2 (127.0.98.3:39831)
I20260925 13:17:51.904913 7690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108151639df24d4dbce46edfa7a9d4d0" candidate_uuid: "0fb1305a6a8a403cb15b4b556c796b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0f8a13eb91545bfa682c88cef4201a2"
I20260925 13:17:51.904919 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108151639df24d4dbce46edfa7a9d4d0" candidate_uuid: "0fb1305a6a8a403cb15b4b556c796b04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62cf9c8eb1f74bdaa5e251667e493263"
I20260925 13:17:51.904981 7554 raft_consensus.cc:3037] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:51.904982 7690 raft_consensus.cc:3037] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:51.905587 7690 raft_consensus.cc:2454] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fb1305a6a8a403cb15b4b556c796b04 in term 1.
I20260925 13:17:51.905635 7554 raft_consensus.cc:2454] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fb1305a6a8a403cb15b4b556c796b04 in term 1.
I20260925 13:17:51.905751 7352 leader_election.cc:302] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fb1305a6a8a403cb15b4b556c796b04, a0f8a13eb91545bfa682c88cef4201a2; no voters:
I20260925 13:17:51.905838 7828 raft_consensus.cc:2781] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:51.905972 7828 raft_consensus.cc:686] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [term 1 LEADER]: Becoming Leader. State: Replica: 0fb1305a6a8a403cb15b4b556c796b04, State: Running, Role: LEADER
I20260925 13:17:51.906057 7828 consensus_queue.cc:237] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } } peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260925 13:17:51.906260 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.906163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:53645)
0925 13:17:51.906195 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.906232 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 13:17:51.906737 7096 catalog_manager.cc:5495] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fb1305a6a8a403cb15b4b556c796b04 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0fb1305a6a8a403cb15b4b556c796b04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1305a6a8a403cb15b4b556c796b04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } health_report { overall_health: UNKNOWN } } }
I20260925 13:17:51.915413 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.913404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.0.98.1:35095)
0925 13:17:51.913440 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:51.913441 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:51.913454 (+ 13us) server_negotiation.cc:407] Connection header received
0925 13:17:51.913527 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:51.913561 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:51.913583 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:51.913642 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:51.913733 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.914178 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.914895 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:51.915071 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:51.915114 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:51.915200 (+ 86us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:51.915293 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:51.915337 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:51.915348 (+ 11us) server_negotiation.cc:299] Negotiation successful
0925 13:17:51.915363 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:17:51.916352 7690 raft_consensus.cc:1261] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2 [term 1 FOLLOWER]: Refusing update from remote peer 0fb1305a6a8a403cb15b4b556c796b04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:51.916406 7554 raft_consensus.cc:1261] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263 [term 1 FOLLOWER]: Refusing update from remote peer 0fb1305a6a8a403cb15b4b556c796b04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:51.916580 7828 consensus_queue.cc:1035] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0f8a13eb91545bfa682c88cef4201a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:51.916698 7839 consensus_queue.cc:1035] T 108151639df24d4dbce46edfa7a9d4d0 P 0fb1305a6a8a403cb15b4b556c796b04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62cf9c8eb1f74bdaa5e251667e493263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:51.917505 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.917363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:60181)
0925 13:17:51.917412 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.917448 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 13:17:51.917738 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.917644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:38373)
0925 13:17:51.917678 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.917711 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 13:17:51.918417 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.918338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:45297)
0925 13:17:51.918371 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.918392 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 13:17:51.918983 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.918538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.3:35215)
0925 13:17:51.918930 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.918959 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378}
I20260925 13:17:51.919409 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.919286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.1:35909)
0925 13:17:51.919363 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.919385 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 13:17:51.920572 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.920480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34171 (local address 127.0.98.2:51551)
0925 13:17:51.920524 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.920547 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 13:17:51.920994 7097 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:51.921065 7097 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:51.922127 7097 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9318f44ac5814b20b91da5f65f9d1f20: Sending DeleteTablet for 3 replicas of tablet 108151639df24d4dbce46edfa7a9d4d0
I20260925 13:17:51.922392 7397 tablet_service.cc:1514] Processing DeleteTablet for tablet 108151639df24d4dbce46edfa7a9d4d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:51 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39288
I20260925 13:17:51.922428 7534 tablet_service.cc:1514] Processing DeleteTablet for tablet 108151639df24d4dbce46edfa7a9d4d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:51 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:52462
I20260925 13:17:51.922569 7670 tablet_service.cc:1514] Processing DeleteTablet for tablet 108151639df24d4dbce46edfa7a9d4d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:51 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49178
I20260925 13:17:51.922729 7854 tablet_replica.cc:330] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2: stopping tablet replica
I20260925 13:17:51.922773 7854 raft_consensus.cc:2229] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:51.922802 7854 raft_consensus.cc:2258] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:51.923206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 7245
I20260925 13:17:51.925799 7854 ts_tablet_manager.cc:1905] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:51.927251 7854 ts_tablet_manager.cc:1918] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:51.927299 7854 log.cc:1196] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/108151639df24d4dbce46edfa7a9d4d0
I20260925 13:17:51.927493 7854 ts_tablet_manager.cc:1939] T 108151639df24d4dbce46edfa7a9d4d0 P a0f8a13eb91545bfa682c88cef4201a2: Deleting consensus metadata
I20260925 13:17:51.927763 7083 catalog_manager.cc:4841] TS a0f8a13eb91545bfa682c88cef4201a2 (127.0.98.3:39831): tablet 108151639df24d4dbce46edfa7a9d4d0 (table test-table [id=7a3e5caace734837852deb7b46e62d5c]) successfully deleted
W20260925 13:17:51.928726 7824 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:51.928595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35095 (local address 127.0.0.1:39298)
0925 13:17:51.928648 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.928677 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35095: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 13:17:51.928802 7853 tablet_replica.cc:330] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263: stopping tablet replica
I20260925 13:17:51.928859 7853 raft_consensus.cc:2229] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 13:17:51.928869 7081 catalog_manager.cc:4584] TS 0fb1305a6a8a403cb15b4b556c796b04 (127.0.98.1:35095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 108151639df24d4dbce46edfa7a9d4d0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35095: connect: Connection reset by peer (error 104)
I20260925 13:17:51.928912 7853 raft_consensus.cc:2258] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:51.929034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 7471
I20260925 13:17:51.929210 7853 ts_tablet_manager.cc:1905] T 108151639df24d4dbce46edfa7a9d4d0 P 62cf9c8eb1f74bdaa5e251667e493263: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:51.935912 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:51.935790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45815 (local address 127.0.0.1:52470)
0925 13:17:51.935841 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:51.935874 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 13:17:51.935959 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 7607
I20260925 13:17:51.939797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 6964
I20260925 13:17:51.943674 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 7151
2026-09-25T13:17:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4426 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:17:51 dist-test-slave-t8d9 krb5kdc[7857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:17:51 dist-test-slave-t8d9 krb5kdc[7857](info): set up 2 sockets
Sep 25 13:17:51 dist-test-slave-t8d9 krb5kdc[7857](info): commencing operation
krb5kdc: starting...
W20260925 13:17:52.913787 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:17:52 dist-test-slave-t8d9 krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:17:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 13:17:53.024137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39083
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:53.102835 7870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:53.103075 7870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:53.103106 7870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:53.104393 7870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:53.104446 7870 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 13:17:53.104463 7870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:53.104480 7870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:53.104494 7870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:53.104511 7870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:53.106129 7870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39083
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:53.106393 7870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:53.106606 7870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:53.111230 7875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:53.111239 7878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:53.111389 7870 server_base.cc:1034] running on GCE node
W20260925 13:17:53.111447 7876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:53.111673 7870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:53.111968 7870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:53.113126 7870 hybrid_clock.cc:648] HybridClock initialized: now 1790342273113105 us; error 32 us; skew 500 ppm
Sep 25 13:17:53 dist-test-slave-t8d9 krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:53.115487 7870 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 13:17:53.115916 7870 webserver.cc:460] Webserver started at http://127.0.98.62:35021/ using document root <none> and password file <none>
I20260925 13:17:53.116103 7870 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:53.116142 7870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:53.116233 7870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:53.117244 7870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "cdd886dfe16f424f9173a574b861b329"
format_stamp: "Formatted at 2026-09-25 13:17:53 on dist-test-slave-t8d9"
server_key: "fb08eb8d48612f66a1a7c8607b2aa961"
server_key_iv: "fee42e556180980882f85c2fac0fb492"
server_key_version: "encryptionkey@0"
I20260925 13:17:53.117563 7870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "cdd886dfe16f424f9173a574b861b329"
format_stamp: "Formatted at 2026-09-25 13:17:53 on dist-test-slave-t8d9"
server_key: "fb08eb8d48612f66a1a7c8607b2aa961"
server_key_iv: "fee42e556180980882f85c2fac0fb492"
server_key_version: "encryptionkey@0"
I20260925 13:17:53.118749 7870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:53.119371 7885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:53.119498 7870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:53.119568 7870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "cdd886dfe16f424f9173a574b861b329"
format_stamp: "Formatted at 2026-09-25 13:17:53 on dist-test-slave-t8d9"
server_key: "fb08eb8d48612f66a1a7c8607b2aa961"
server_key_iv: "fee42e556180980882f85c2fac0fb492"
server_key_version: "encryptionkey@0"
I20260925 13:17:53.119619 7870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:53.131971 7870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:53.133776 7870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:53.133889 7870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:53.138432 7870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39083
I20260925 13:17:53.138497 7947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39083 every 8 connection(s)
I20260925 13:17:53.138811 7870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:53.139451 7948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:53.141770 7948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329: Bootstrap starting.
I20260925 13:17:53.142328 7948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:53.142603 7948 log.cc:824] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:53.143227 7948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329: No bootstrap required, opened a new log
I20260925 13:17:53.145255 7948 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd886dfe16f424f9173a574b861b329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39083 } }
I20260925 13:17:53.145411 7948 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:53.145431 7948 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd886dfe16f424f9173a574b861b329, State: Initialized, Role: FOLLOWER
I20260925 13:17:53.145633 7948 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd886dfe16f424f9173a574b861b329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39083 } }
I20260925 13:17:53.145713 7948 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:53.145738 7948 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:53.145784 7948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:53.146306 7948 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd886dfe16f424f9173a574b861b329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39083 } }
I20260925 13:17:53.146394 7948 leader_election.cc:302] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdd886dfe16f424f9173a574b861b329; no voters:
I20260925 13:17:53.146556 7948 leader_election.cc:288] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:53.146600 7951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:53.146757 7951 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [term 1 LEADER]: Becoming Leader. State: Replica: cdd886dfe16f424f9173a574b861b329, State: Running, Role: LEADER
I20260925 13:17:53.146801 7948 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:53.146849 7951 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd886dfe16f424f9173a574b861b329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39083 } }
I20260925 13:17:53.147186 7952 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdd886dfe16f424f9173a574b861b329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd886dfe16f424f9173a574b861b329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39083 } } }
I20260925 13:17:53.147195 7953 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd886dfe16f424f9173a574b861b329. Latest consensus state: current_term: 1 leader_uuid: "cdd886dfe16f424f9173a574b861b329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd886dfe16f424f9173a574b861b329" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39083 } } }
I20260925 13:17:53.147378 7953 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:53.147562 7952 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:53.147709 7960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:53.148219 7960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:53.148628 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 7870
I20260925 13:17:53.148736 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:53.148933 392 external_mini_cluster.cc:1428] Setting key d122c1a7624b054c8b8de24a5100834b
I20260925 13:17:53.149935 7960 catalog_manager.cc:1347] Generated new cluster ID: bb378bc7275d4362879f53b631a4c21f
I20260925 13:17:53.149984 7960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:17:53 dist-test-slave-t8d9 krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:17:53.160274 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.149608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57846 (local address 127.0.98.62:39083)
0925 13:17:53.149756 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.149765 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.149798 (+ 33us) server_negotiation.cc:407] Connection header received
0925 13:17:53.149919 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.149927 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.149984 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.150141 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:53.150795 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.152264 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.153284 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.154024 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.154141 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.156814 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:53.156819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:53.156819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:53.156847 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:53.158707 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.159103 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.159106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.159107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.159205 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.159378 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.159379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.159379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.159669 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:53.159782 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:53.159896 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 13:17:53.160035 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260925 13:17:53.162251 7960 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:53.162945 7960 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:53.166811 7960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329: Generated new TSK 0
I20260925 13:17:53.166985 7960 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 13:17:53.192170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39083
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:53.273190 7974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:53.273414 7974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:53.273439 7974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:53.274715 7974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:53.274771 7974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:53.274792 7974 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:17:53.274809 7974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:17:53.276397 7974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:53.276682 7974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:53.276911 7974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:53.281916 7980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:53.281945 7982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:53.282006 7974 server_base.cc:1034] running on GCE node
W20260925 13:17:53.281910 7979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:53.282318 7974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:53.282584 7974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:53.283739 7974 hybrid_clock.cc:648] HybridClock initialized: now 1790342273283723 us; error 27 us; skew 500 ppm
Sep 25 13:17:53 dist-test-slave-t8d9 krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:53.286283 7974 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 13:17:53.286762 7974 webserver.cc:460] Webserver started at http://127.0.98.1:36035/ using document root <none> and password file <none>
I20260925 13:17:53.286991 7974 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:53.287045 7974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:53.287174 7974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:53.288247 7974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "69153cd48ea44695a251033a9e00cdeb"
format_stamp: "Formatted at 2026-09-25 13:17:53 on dist-test-slave-t8d9"
server_key: "8f4caa4c53d0116069e370503c934d7f"
server_key_iv: "8ef405c88548b8e1ea5872f8de8181db"
server_key_version: "encryptionkey@0"
I20260925 13:17:53.288586 7974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "69153cd48ea44695a251033a9e00cdeb"
format_stamp: "Formatted at 2026-09-25 13:17:53 on dist-test-slave-t8d9"
server_key: "8f4caa4c53d0116069e370503c934d7f"
server_key_iv: "8ef405c88548b8e1ea5872f8de8181db"
server_key_version: "encryptionkey@0"
I20260925 13:17:53.290032 7974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:17:53.290822 7989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:53.291049 7974 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:17:53.291169 7974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "69153cd48ea44695a251033a9e00cdeb"
format_stamp: "Formatted at 2026-09-25 13:17:53 on dist-test-slave-t8d9"
server_key: "8f4caa4c53d0116069e370503c934d7f"
server_key_iv: "8ef405c88548b8e1ea5872f8de8181db"
server_key_version: "encryptionkey@0"
I20260925 13:17:53.291258 7974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:53.318722 7974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:53.320891 7974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:53.321027 7974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:53.321517 7974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:53.321574 7974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:53.321622 7974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:53.321645 7974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:53.328490 7974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33769
I20260925 13:17:53.328600 8104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33769 every 8 connection(s)
Sep 25 13:17:53 dist-test-slave-t8d9 krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:17:53.328872 7974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:17:53.331645 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.322045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41285 (local address 127.0.98.62:39083)
0925 13:17:53.322106 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.322109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.322321 (+ 212us) server_negotiation.cc:407] Connection header received
0925 13:17:53.322991 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.322992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.323005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.323062 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:53.323984 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.324756 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.326338 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.326660 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.326724 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.329345 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:53.329347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:53.329347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:53.329355 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:53.330447 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.330797 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.330797 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.330798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.330843 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.331121 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.331122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.331122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.331217 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:53.331259 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:53.331501 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 13:17:53.331566 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:17:53.331835 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.322014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39083 (local address 127.0.98.1:41285)
0925 13:17:53.322189 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:53.322238 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:53.322890 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:53.323179 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:53.323183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:53.323536 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:53.323866 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.323873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.324868 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.324869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.326252 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.326256 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.326744 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.326745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.326902 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.327516 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:53.327529 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:53.329257 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:53.330513 (+ 1256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:53.330519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:53.330522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:53.330736 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:53.330930 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:53.330931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:53.330931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:53.331059 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:53.331268 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:53.331271 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:53.331397 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 13:17:53.331484 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 13:17:53.331619 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260925 13:17:53.332378 8105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39083
I20260925 13:17:53.332458 8105 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:53.332619 8105 heartbeater.cc:511] Master 127.0.98.62:39083 requested a full tablet report, sending...
I20260925 13:17:53.333580 7901 ts_manager.cc:194] Registered new tserver with Master: 69153cd48ea44695a251033a9e00cdeb (127.0.98.1:33769)
I20260925 13:17:53.334334 7901 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41285
I20260925 13:17:53.337347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 7974
I20260925 13:17:53.337447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:17:53.337654 392 external_mini_cluster.cc:1428] Setting key a566806679fa3b4a43c95a7a16b96755
I20260925 13:17:53.339249 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.333363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59761 (local address 127.0.98.62:39083)
0925 13:17:53.333413 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.333415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.333509 (+ 94us) server_negotiation.cc:407] Connection header received
0925 13:17:53.333602 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.333602 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.333613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.333665 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:53.334073 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.334632 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.335656 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.335839 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.335896 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.336798 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:53.336799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:53.336800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:53.336807 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:53.338225 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.338610 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.338611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.338611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.338671 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.338855 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.338855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.338855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.338947 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:53.338990 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:53.339127 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 13:17:53.339187 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:17:53.339313 8111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.333305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39083 (local address 127.0.98.1:59761)
0925 13:17:53.333435 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:53.333446 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:53.333507 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:53.333700 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:53.333701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:53.333927 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:53.333990 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.333992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.335101 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.335101 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.335572 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.335573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.335948 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.335949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.336021 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.336438 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:53.336443 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:53.336725 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:53.338327 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:53.338328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:53.338328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:53.338546 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:53.338731 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:53.338732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:53.338732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:53.338800 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:53.338995 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:53.338995 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:53.339044 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 13:17:53.339111 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 13:17:53.339210 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 13:17:53.362178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39083
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:53.441499 8114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:53.441772 8114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:53.441797 8114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:53.442981 8114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:53.443030 8114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:53.443049 8114 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:17:53.443069 8114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:17:53.444576 8114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:53.444801 8114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:53.445015 8114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:53.449831 8120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:53.449880 8122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:53.449942 8119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:53.450143 8114 server_base.cc:1034] running on GCE node
I20260925 13:17:53.450316 8114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:53.450527 8114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:53.451712 8114 hybrid_clock.cc:648] HybridClock initialized: now 1790342273451696 us; error 29 us; skew 500 ppm
Sep 25 13:17:53 dist-test-slave-t8d9 krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:53.454252 8114 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 13:17:53.454654 8114 webserver.cc:460] Webserver started at http://127.0.98.2:42679/ using document root <none> and password file <none>
I20260925 13:17:53.454841 8114 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:53.454880 8114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:53.454980 8114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:53.455967 8114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "e5e6a16415d5473cb8005586f8f0652d"
format_stamp: "Formatted at 2026-09-25 13:17:53 on dist-test-slave-t8d9"
server_key: "013af18adbacdd3adeaad0de341b62a0"
server_key_iv: "f875511b192f6d0b780f5d5ca9f86c2b"
server_key_version: "encryptionkey@0"
I20260925 13:17:53.456251 8114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e5e6a16415d5473cb8005586f8f0652d"
format_stamp: "Formatted at 2026-09-25 13:17:53 on dist-test-slave-t8d9"
server_key: "013af18adbacdd3adeaad0de341b62a0"
server_key_iv: "f875511b192f6d0b780f5d5ca9f86c2b"
server_key_version: "encryptionkey@0"
I20260925 13:17:53.457495 8114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:53.458238 8129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:53.458470 8114 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:53.458561 8114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "e5e6a16415d5473cb8005586f8f0652d"
format_stamp: "Formatted at 2026-09-25 13:17:53 on dist-test-slave-t8d9"
server_key: "013af18adbacdd3adeaad0de341b62a0"
server_key_iv: "f875511b192f6d0b780f5d5ca9f86c2b"
server_key_version: "encryptionkey@0"
I20260925 13:17:53.458629 8114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:53.478148 8114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:53.480162 8114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:53.480268 8114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:53.480721 8114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:53.480764 8114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:53.480790 8114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:53.480803 8114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:53.487020 8114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36055
I20260925 13:17:53.487085 8244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36055 every 8 connection(s)
I20260925 13:17:53.487430 8114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:17:53 dist-test-slave-t8d9 krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:17:53.490857 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.481258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58443 (local address 127.0.98.62:39083)
0925 13:17:53.481302 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.481304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.481552 (+ 248us) server_negotiation.cc:407] Connection header received
0925 13:17:53.482150 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.482151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.482166 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.482211 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:53.483058 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.483833 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.485313 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.485640 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.485718 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.488221 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:53.488223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:53.488223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:53.488232 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:53.489425 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.489868 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.489870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.489870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.489942 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.490179 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.490180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.490180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.490310 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:53.490388 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:53.490667 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 13:17:53.490760 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:17:53.491030 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.481267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39083 (local address 127.0.98.2:58443)
0925 13:17:53.481424 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:53.481461 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:53.482036 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:53.482294 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:53.482300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:53.482644 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:53.482946 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.482953 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.483936 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.483937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.485227 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.485230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.485732 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.485733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.485878 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.486446 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:53.486457 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:53.488129 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:53.489522 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:53.489529 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:53.489556 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:53.489790 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:53.490004 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:53.490005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:53.490005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:53.490117 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:53.490370 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:53.490376 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:53.490531 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 13:17:53.490635 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 13:17:53.490800 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260925 13:17:53.491575 8245 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39083
I20260925 13:17:53.491659 8245 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:53.491847 8245 heartbeater.cc:511] Master 127.0.98.62:39083 requested a full tablet report, sending...
I20260925 13:17:53.492273 7901 ts_manager.cc:194] Registered new tserver with Master: e5e6a16415d5473cb8005586f8f0652d (127.0.98.2:36055)
I20260925 13:17:53.492758 7901 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58443
I20260925 13:17:53.497110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 8114
I20260925 13:17:53.497224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:17:53.497442 392 external_mini_cluster.cc:1428] Setting key 2b10dba0f186f710f480faf41e31488a
I20260925 13:17:53.497890 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.492657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60023 (local address 127.0.98.62:39083)
0925 13:17:53.492708 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.492709 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.492756 (+ 47us) server_negotiation.cc:407] Connection header received
0925 13:17:53.492811 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.492812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.492822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.492875 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:53.493255 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.493852 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.494548 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.494723 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.494769 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.495676 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:53.495677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:53.495678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:53.495683 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:53.496835 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.497113 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.497114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.497114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.497170 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.497348 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.497349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.497349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.497557 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:53.497622 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:53.497756 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 13:17:53.497831 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:17:53.497933 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.492582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39083 (local address 127.0.98.2:60023)
0925 13:17:53.492699 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:53.492709 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:53.492768 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:53.492920 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:53.492921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:53.493121 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:53.493183 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.493184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.493927 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.493928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.494479 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.494481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.494835 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.494836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.494904 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.495321 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:53.495325 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:53.495610 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:53.496895 (+ 1285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:53.496895 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:53.496896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:53.497051 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:53.497232 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:53.497233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:53.497233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:53.497292 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:53.497614 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:53.497615 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:53.497655 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 13:17:53.497729 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 13:17:53.497831 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 13:17:53.520349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39083
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:53.598712 8254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:53.598915 8254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:53.598941 8254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:53.600071 8254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:53.600112 8254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:53.600131 8254 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 13:17:53.600148 8254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:17:53.601630 8254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:53.601862 8254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:53.602070 8254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:53.606775 8262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:53.606896 8254 server_base.cc:1034] running on GCE node
W20260925 13:17:53.606781 8259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:53.606978 8260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:53.607164 8254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:53.607388 8254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:53.608528 8254 hybrid_clock.cc:648] HybridClock initialized: now 1790342273608515 us; error 29 us; skew 500 ppm
Sep 25 13:17:53 dist-test-slave-t8d9 krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:53.610924 8254 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 13:17:53.611367 8254 webserver.cc:460] Webserver started at http://127.0.98.3:33893/ using document root <none> and password file <none>
I20260925 13:17:53.611579 8254 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:53.611629 8254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:53.611752 8254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:53.612771 8254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "76b4e68537f845c0ba85219786abd160"
format_stamp: "Formatted at 2026-09-25 13:17:53 on dist-test-slave-t8d9"
server_key: "a63047098054d1fefb2ab47878c72c8f"
server_key_iv: "e268b148eab31367e396b7789860fcea"
server_key_version: "encryptionkey@0"
I20260925 13:17:53.613112 8254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "76b4e68537f845c0ba85219786abd160"
format_stamp: "Formatted at 2026-09-25 13:17:53 on dist-test-slave-t8d9"
server_key: "a63047098054d1fefb2ab47878c72c8f"
server_key_iv: "e268b148eab31367e396b7789860fcea"
server_key_version: "encryptionkey@0"
I20260925 13:17:53.614418 8254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:53.615155 8269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:53.615307 8254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:53.615386 8254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "76b4e68537f845c0ba85219786abd160"
format_stamp: "Formatted at 2026-09-25 13:17:53 on dist-test-slave-t8d9"
server_key: "a63047098054d1fefb2ab47878c72c8f"
server_key_iv: "e268b148eab31367e396b7789860fcea"
server_key_version: "encryptionkey@0"
I20260925 13:17:53.615442 8254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:53.625049 8254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:53.626976 8254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:53.627125 8254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:53.627581 8254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:53.627620 8254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:53.627650 8254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:53.627666 8254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 13:17:53 dist-test-slave-t8d9 krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:17:53.634207 8254 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37949
I20260925 13:17:53.634532 8254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:17:53.634558 8384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37949 every 8 connection(s)
I20260925 13:17:53.635061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 8254
I20260925 13:17:53.635183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:17:53.635407 392 external_mini_cluster.cc:1428] Setting key 8c1a6d23aa7efbd4d1009e5252ed06a5
I20260925 13:17:53.636968 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.628083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39361 (local address 127.0.98.62:39083)
0925 13:17:53.628148 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.628150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.628322 (+ 172us) server_negotiation.cc:407] Connection header received
0925 13:17:53.628818 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.628820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.628833 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.628898 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:53.629749 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.630480 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.631497 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.631703 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.631767 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.633998 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:53.634000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:53.634000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:53.634008 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:53.635647 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.636044 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.636046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.636046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.636109 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.636332 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.636333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.636333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.636454 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:53.636517 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:53.636779 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 13:17:53.636864 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:17:53.637112 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.628049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39083 (local address 127.0.98.3:39361)
0925 13:17:53.628200 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:53.628239 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:53.628726 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:53.628967 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:53.628971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:53.629302 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:53.629634 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.629641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.630578 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.630579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.631405 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.631407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.631793 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.631794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.631957 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.632440 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:53.632450 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:53.633903 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:53.635732 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:53.635738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:53.635739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:53.635971 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:53.636166 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:53.636167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:53.636168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:53.636273 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:53.636513 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:53.636520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:53.636656 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 13:17:53.636750 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 13:17:53.636907 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260925 13:17:53.637581 8385 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39083
I20260925 13:17:53.637665 8385 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:53.637826 8385 heartbeater.cc:511] Master 127.0.98.62:39083 requested a full tablet report, sending...
I20260925 13:17:53.638219 7902 ts_manager.cc:194] Registered new tserver with Master: 76b4e68537f845c0ba85219786abd160 (127.0.98.3:37949)
I20260925 13:17:53.638675 7902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39361
I20260925 13:17:53.643249 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.638571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34471 (local address 127.0.98.62:39083)
0925 13:17:53.638610 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.638611 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.638659 (+ 48us) server_negotiation.cc:407] Connection header received
0925 13:17:53.638714 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.638714 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.638723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.638768 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:53.639083 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.639536 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.640200 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.640394 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.640444 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.641216 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:53.641217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:53.641218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:53.641224 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:53.642296 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.642579 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.642580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.642581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.642629 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.642780 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.642781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.642781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.642882 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:53.642936 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:53.643085 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 13:17:53.643157 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:17:53.643267 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.638497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39083 (local address 127.0.98.3:34471)
0925 13:17:53.638613 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:53.638623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:53.638673 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:53.638772 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:53.638772 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:53.638955 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:53.639009 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.639011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.639622 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.639623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.640132 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.640133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.640447 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.640448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.640509 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.640882 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:53.640885 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:53.641143 (+ 258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:53.642363 (+ 1220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:53.642364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:53.642364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:53.642515 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:53.642679 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:53.642680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:53.642680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:53.642733 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:53.642933 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:53.642934 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:53.642977 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 13:17:53.643061 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 13:17:53.643157 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260925 13:17:53.646988 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 13:17:53 dist-test-slave-t8d9 krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342273, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 13:17:53 dist-test-slave-t8d9 krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:53 dist-test-slave-t8d9 krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 13:17:53.667407 7971 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:53.661784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.0.98.62:39083)
0925 13:17:53.661843 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.661845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.661854 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:53.661880 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.661881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.661891 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.661940 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:53.662468 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.662958 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.663950 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.664230 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.664285 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.667323 (+ 3038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57850: BlockingRecv error: recv got EOF from 127.0.0.1:57850 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 13:17:53 dist-test-slave-t8d9 krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342273, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:17:53.674117 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.668020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.0.98.62:39083)
0925 13:17:53.668080 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.668082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.668162 (+ 80us) server_negotiation.cc:407] Connection header received
0925 13:17:53.668229 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.668230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.668240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.668285 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:53.668715 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.669187 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.669866 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.670062 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.670114 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.671985 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:53.671987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:53.671988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:53.671995 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:53.673120 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.673405 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.673406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.673407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.673457 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.673656 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.673657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.673657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.673765 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:53.673853 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:53.673946 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 13:17:53.674023 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 13:17:53.675557 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:17:53.677047 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:53.683426 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.679187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37949 (local address 127.0.0.1:57082)
0925 13:17:53.679347 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:53.679387 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:53.679447 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:53.679831 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:53.679832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:53.679833 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:53.679956 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.679959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.681231 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.681231 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.682279 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.682281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.683138 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.683139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.683212 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.683231 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 13:17:53.683274 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:17:53.683326 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":39,"thread_start_us":44,"threads_started":1}
I20260925 13:17:53.683447 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.679572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.0.98.1:33769)
0925 13:17:53.679718 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.679723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.679772 (+ 49us) server_negotiation.cc:407] Connection header received
0925 13:17:53.679840 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.679842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.679856 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.679931 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:53.680158 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.680876 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.682303 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.683051 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.683125 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.683205 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:53.683311 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 13:17:53.683352 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260925 13:17:53.683422 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.679361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33769 (local address 127.0.0.1:47888)
0925 13:17:53.679721 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:53.679730 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:53.679781 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:53.680040 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:53.680041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:53.680043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:53.680085 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.680086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.680952 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.680953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.682234 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.682237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.683124 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.683124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.683213 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.683228 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 13:17:53.683293 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 13:17:53.683340 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":71,"threads_started":1}
I20260925 13:17:53.683734 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.679318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.0.98.3:37949)
0925 13:17:53.679680 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.679684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.679693 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:53.679726 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.679728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.679744 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.679901 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:53.680036 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.681143 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.682342 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.683051 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.683469 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.683558 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:53.683584 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 13:17:53.683644 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":68,"threads_started":1}
I20260925 13:17:53.684273 8039 tablet_service.cc:1467] Processing CreateTablet for tablet 0969bb08f68145c9a501e5b65ba2af84 (DEFAULT_TABLE table=test-table [id=087ff7eff70642d0873028a07314a0bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:53.684327 8319 tablet_service.cc:1467] Processing CreateTablet for tablet 0969bb08f68145c9a501e5b65ba2af84 (DEFAULT_TABLE table=test-table [id=087ff7eff70642d0873028a07314a0bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:53.684641 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.679203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36055 (local address 127.0.0.1:51886)
0925 13:17:53.679348 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:53.679389 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:53.679448 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:53.679827 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:53.679830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:53.679832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:53.679930 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.679932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.682358 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.682359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.683002 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.683003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.684437 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.684438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.684509 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.684513 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:53.684541 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 13:17:53.684579 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260925 13:17:53.684649 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0969bb08f68145c9a501e5b65ba2af84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:53.684664 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0969bb08f68145c9a501e5b65ba2af84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:53.684993 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.679491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.0.98.2:36055)
0925 13:17:53.679648 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.679653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.679664 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:17:53.679704 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.679706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.679721 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.680016 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:53.680076 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.681145 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.683084 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.684339 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.684739 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.684834 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:53.684861 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 13:17:53.684906 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260925 13:17:53.685678 8179 tablet_service.cc:1467] Processing CreateTablet for tablet 0969bb08f68145c9a501e5b65ba2af84 (DEFAULT_TABLE table=test-table [id=087ff7eff70642d0873028a07314a0bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:53.685989 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0969bb08f68145c9a501e5b65ba2af84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:53.687849 8413 tablet_bootstrap.cc:492] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb: Bootstrap starting.
I20260925 13:17:53.688467 8413 tablet_bootstrap.cc:654] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:53.688737 8413 log.cc:824] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:53.689407 8413 tablet_bootstrap.cc:492] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb: No bootstrap required, opened a new log
I20260925 13:17:53.689429 8416 tablet_bootstrap.cc:492] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d: Bootstrap starting.
I20260925 13:17:53.689469 8413 ts_tablet_manager.cc:1397] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:17:53.689524 8414 tablet_bootstrap.cc:492] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160: Bootstrap starting.
I20260925 13:17:53.690056 8416 tablet_bootstrap.cc:654] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:53.690080 8414 tablet_bootstrap.cc:654] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:53.690382 8414 log.cc:824] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:53.690382 8416 log.cc:824] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:53.691045 8416 tablet_bootstrap.cc:492] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d: No bootstrap required, opened a new log
I20260925 13:17:53.691103 8416 ts_tablet_manager.cc:1397] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:17:53.691140 8414 tablet_bootstrap.cc:492] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160: No bootstrap required, opened a new log
I20260925 13:17:53.691211 8414 ts_tablet_manager.cc:1397] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 13:17:53.691560 8413 raft_consensus.cc:348] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } }
I20260925 13:17:53.691737 8413 raft_consensus.cc:374] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:53.691769 8413 raft_consensus.cc:729] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69153cd48ea44695a251033a9e00cdeb, State: Initialized, Role: FOLLOWER
I20260925 13:17:53.691979 8413 consensus_queue.cc:260] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } }
I20260925 13:17:53.692302 8105 heartbeater.cc:503] Master 127.0.98.62:39083 was elected leader, sending a full tablet report...
I20260925 13:17:53.692554 8413 ts_tablet_manager.cc:1428] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 13:17:53.692839 8416 raft_consensus.cc:348] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } }
I20260925 13:17:53.692966 8416 raft_consensus.cc:374] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:53.692996 8416 raft_consensus.cc:729] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e6a16415d5473cb8005586f8f0652d, State: Initialized, Role: FOLLOWER
I20260925 13:17:53.693115 8416 consensus_queue.cc:260] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } }
I20260925 13:17:53.693372 8416 ts_tablet_manager.cc:1428] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:17:53.693418 8245 heartbeater.cc:503] Master 127.0.98.62:39083 was elected leader, sending a full tablet report...
I20260925 13:17:53.693356 8414 raft_consensus.cc:348] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } }
I20260925 13:17:53.693579 8414 raft_consensus.cc:374] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:53.693624 8414 raft_consensus.cc:729] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b4e68537f845c0ba85219786abd160, State: Initialized, Role: FOLLOWER
I20260925 13:17:53.693735 8414 consensus_queue.cc:260] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } }
I20260925 13:17:53.693964 8414 ts_tablet_manager.cc:1428] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 13:17:53.694000 8385 heartbeater.cc:503] Master 127.0.98.62:39083 was elected leader, sending a full tablet report...
I20260925 13:17:53.732600 8420 raft_consensus.cc:482] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:53.732756 8420 raft_consensus.cc:504] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } }
I20260925 13:17:53.733074 8420 leader_election.cc:288] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76b4e68537f845c0ba85219786abd160 (127.0.98.3:37949), 69153cd48ea44695a251033a9e00cdeb (127.0.98.1:33769)
I20260925 13:17:53.735759 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.733360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46255 (local address 127.0.98.3:37949)
0925 13:17:53.733424 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.733426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.733459 (+ 33us) server_negotiation.cc:407] Connection header received
0925 13:17:53.733489 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.733490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.733500 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.733571 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:53.733727 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.734203 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.735103 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.735441 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.735498 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.735508 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:53.735660 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 13:17:53.735694 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 13:17:53.735774 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.733294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37949 (local address 127.0.98.2:46255)
0925 13:17:53.733347 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:53.733360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:53.733422 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:53.733601 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:53.733601 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:53.733604 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:53.733668 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.733670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.734275 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.734276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.735020 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.735022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.735507 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.735508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.735602 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.735605 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:53.735649 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 13:17:53.735687 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 13:17:53.735848 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.733505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38113 (local address 127.0.98.1:33769)
0925 13:17:53.733565 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.733567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.733660 (+ 93us) server_negotiation.cc:407] Connection header received
0925 13:17:53.733704 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.733705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.733715 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.733760 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:53.733881 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.734346 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.735208 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.735559 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.735616 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.735625 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:53.735748 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 13:17:53.735783 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 13:17:53.735864 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.733399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33769 (local address 127.0.98.2:38113)
0925 13:17:53.733603 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:53.733611 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:53.733655 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:53.733762 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:53.733762 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:53.733764 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:53.733828 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.733830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.734416 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.734416 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.735128 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.735130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.735626 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.735627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.735691 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.735694 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:53.735737 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:17:53.735774 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":120,"threads_started":1}
I20260925 13:17:53.736142 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0969bb08f68145c9a501e5b65ba2af84" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b4e68537f845c0ba85219786abd160" is_pre_election: true
I20260925 13:17:53.736138 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0969bb08f68145c9a501e5b65ba2af84" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69153cd48ea44695a251033a9e00cdeb" is_pre_election: true
I20260925 13:17:53.736282 8059 raft_consensus.cc:2454] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 0.
I20260925 13:17:53.736282 8339 raft_consensus.cc:2454] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 0.
I20260925 13:17:53.736495 8132 leader_election.cc:302] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69153cd48ea44695a251033a9e00cdeb, e5e6a16415d5473cb8005586f8f0652d; no voters:
I20260925 13:17:53.736619 8420 raft_consensus.cc:2781] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:53.736688 8420 raft_consensus.cc:482] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:53.736712 8420 raft_consensus.cc:3037] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:53.737773 8420 raft_consensus.cc:504] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } }
I20260925 13:17:53.737875 8420 leader_election.cc:288] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 election: Requested vote from peers 76b4e68537f845c0ba85219786abd160 (127.0.98.3:37949), 69153cd48ea44695a251033a9e00cdeb (127.0.98.1:33769)
W20260925 13:17:53.738058 8246 tablet.cc:2369] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:17:53.738060 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0969bb08f68145c9a501e5b65ba2af84" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b4e68537f845c0ba85219786abd160"
I20260925 13:17:53.738099 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0969bb08f68145c9a501e5b65ba2af84" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69153cd48ea44695a251033a9e00cdeb"
I20260925 13:17:53.738152 8339 raft_consensus.cc:3037] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:53.738190 8059 raft_consensus.cc:3037] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:53.738843 8059 raft_consensus.cc:2454] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 1.
I20260925 13:17:53.738900 8339 raft_consensus.cc:2454] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 1.
I20260925 13:17:53.738988 8132 leader_election.cc:302] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69153cd48ea44695a251033a9e00cdeb, e5e6a16415d5473cb8005586f8f0652d; no voters:
I20260925 13:17:53.739109 8420 raft_consensus.cc:2781] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:53.739249 8420 raft_consensus.cc:686] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [term 1 LEADER]: Becoming Leader. State: Replica: e5e6a16415d5473cb8005586f8f0652d, State: Running, Role: LEADER
I20260925 13:17:53.739336 8420 consensus_queue.cc:237] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } }
I20260925 13:17:53.740103 7901 catalog_manager.cc:5495] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d reported cstate change: term changed from 0 to 1, leader changed from <none> to e5e6a16415d5473cb8005586f8f0652d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e5e6a16415d5473cb8005586f8f0652d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } health_report { overall_health: HEALTHY } } }
I20260925 13:17:53.755991 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.753935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.0.98.2:36055)
0925 13:17:53.753986 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.753988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.754014 (+ 26us) server_negotiation.cc:407] Connection header received
0925 13:17:53.754054 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.754055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.754075 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.754125 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:53.754232 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.754702 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.755401 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.755582 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.755623 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.755729 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:53.755833 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:53.755884 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:53.755899 (+ 15us) server_negotiation.cc:299] Negotiation successful
0925 13:17:53.755916 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 13:17:53.757681 8059 raft_consensus.cc:1261] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Refusing update from remote peer e5e6a16415d5473cb8005586f8f0652d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:53.757681 8339 raft_consensus.cc:1261] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Refusing update from remote peer e5e6a16415d5473cb8005586f8f0652d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:53.758024 8420 consensus_queue.cc:1035] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:53.758184 8423 consensus_queue.cc:1035] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:53.760931 8427 mvcc.cc:205] Tried to move back new op lower bound from 7333241953308925952 to 7333241953236942848. Current Snapshot: MvccSnapshot[applied={T|T < 7333241953308925952}]
I20260925 13:17:53.762878 7902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:53.762964 7902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:53.763654 7902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329: Sending DeleteTablet for 3 replicas of tablet 0969bb08f68145c9a501e5b65ba2af84
I20260925 13:17:53.763938 8319 tablet_service.cc:1514] Processing DeleteTablet for tablet 0969bb08f68145c9a501e5b65ba2af84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57082
I20260925 13:17:53.763960 8039 tablet_service.cc:1514] Processing DeleteTablet for tablet 0969bb08f68145c9a501e5b65ba2af84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47888
I20260925 13:17:53.763994 8179 tablet_service.cc:1514] Processing DeleteTablet for tablet 0969bb08f68145c9a501e5b65ba2af84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51886
I20260925 13:17:53.764204 8436 tablet_replica.cc:330] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb: stopping tablet replica
I20260925 13:17:53.764156 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 13:17:53.764273 8436 raft_consensus.cc:2229] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 13:17:53.764287 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:53.764577 8436 raft_consensus.cc:2258] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:53.764743 8435 tablet_replica.cc:330] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160: stopping tablet replica
I20260925 13:17:53.765019 8435 raft_consensus.cc:2229] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:53.765142 8435 raft_consensus.cc:2258] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:53.765471 8039 tablet_service.cc:1467] Processing CreateTablet for tablet a42e09b79edb4c79909539ed77c679d8 (DEFAULT_TABLE table=test-table [id=7092ae5633c643bd8a0ce861377f41bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:53.765475 8179 tablet_service.cc:1467] Processing CreateTablet for tablet a42e09b79edb4c79909539ed77c679d8 (DEFAULT_TABLE table=test-table [id=7092ae5633c643bd8a0ce861377f41bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:53.765619 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42e09b79edb4c79909539ed77c679d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:53.765667 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42e09b79edb4c79909539ed77c679d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:53.765666 8319 tablet_service.cc:1467] Processing CreateTablet for tablet a42e09b79edb4c79909539ed77c679d8 (DEFAULT_TABLE table=test-table [id=7092ae5633c643bd8a0ce861377f41bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:53.765727 8435 ts_tablet_manager.cc:1905] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:53.765749 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42e09b79edb4c79909539ed77c679d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:53.765872 8437 tablet_replica.cc:330] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d: stopping tablet replica
I20260925 13:17:53.766964 8416 tablet_bootstrap.cc:492] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d: Bootstrap starting.
I20260925 13:17:53.767096 8413 tablet_bootstrap.cc:492] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb: Bootstrap starting.
I20260925 13:17:53.767347 8416 tablet_bootstrap.cc:654] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:53.767493 8413 tablet_bootstrap.cc:654] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:53.767688 8414 tablet_bootstrap.cc:492] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160: Bootstrap starting.
I20260925 13:17:53.767865 8435 ts_tablet_manager.cc:1918] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:53.767961 8435 log.cc:1196] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/0969bb08f68145c9a501e5b65ba2af84
I20260925 13:17:53.767961 8416 tablet_bootstrap.cc:492] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d: No bootstrap required, opened a new log
I20260925 13:17:53.768002 8416 ts_tablet_manager.cc:1397] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:53.768100 8414 tablet_bootstrap.cc:654] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:53.768134 8416 raft_consensus.cc:348] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:53.768211 8416 raft_consensus.cc:374] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:53.768206 8435 ts_tablet_manager.cc:1939] T 0969bb08f68145c9a501e5b65ba2af84 P 76b4e68537f845c0ba85219786abd160: Deleting consensus metadata
I20260925 13:17:53.768236 8416 raft_consensus.cc:729] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e6a16415d5473cb8005586f8f0652d, State: Initialized, Role: FOLLOWER
I20260925 13:17:53.768275 8416 consensus_queue.cc:260] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:53.768373 8416 ts_tablet_manager.cc:1428] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:53.768635 8413 tablet_bootstrap.cc:492] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb: No bootstrap required, opened a new log
I20260925 13:17:53.768678 8413 ts_tablet_manager.cc:1397] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 13:17:53.768697 7888 catalog_manager.cc:4841] TS 76b4e68537f845c0ba85219786abd160 (127.0.98.3:37949): tablet 0969bb08f68145c9a501e5b65ba2af84 (table test-table [id=087ff7eff70642d0873028a07314a0bd]) successfully deleted
I20260925 13:17:53.768810 8413 raft_consensus.cc:348] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:53.768879 8413 raft_consensus.cc:374] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:53.768903 8413 raft_consensus.cc:729] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69153cd48ea44695a251033a9e00cdeb, State: Initialized, Role: FOLLOWER
I20260925 13:17:53.768945 8413 consensus_queue.cc:260] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:53.769027 8413 ts_tablet_manager.cc:1428] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:53.769251 8437 raft_consensus.cc:2229] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:17:53.769263 8436 ts_tablet_manager.cc:1905] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:53.769373 8437 raft_consensus.cc:2258] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:53.769814 8437 ts_tablet_manager.cc:1905] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:53.770793 8436 ts_tablet_manager.cc:1918] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:53.770864 8436 log.cc:1196] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/wal/wals/0969bb08f68145c9a501e5b65ba2af84
I20260925 13:17:53.771113 8436 ts_tablet_manager.cc:1939] T 0969bb08f68145c9a501e5b65ba2af84 P 69153cd48ea44695a251033a9e00cdeb: Deleting consensus metadata
I20260925 13:17:53.771272 8437 ts_tablet_manager.cc:1918] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:53.771339 8437 log.cc:1196] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/0969bb08f68145c9a501e5b65ba2af84
I20260925 13:17:53.771513 7888 catalog_manager.cc:4841] TS 69153cd48ea44695a251033a9e00cdeb (127.0.98.1:33769): tablet 0969bb08f68145c9a501e5b65ba2af84 (table test-table [id=087ff7eff70642d0873028a07314a0bd]) successfully deleted
I20260925 13:17:53.771566 8437 ts_tablet_manager.cc:1939] T 0969bb08f68145c9a501e5b65ba2af84 P e5e6a16415d5473cb8005586f8f0652d: Deleting consensus metadata
I20260925 13:17:53.771976 7887 catalog_manager.cc:4841] TS e5e6a16415d5473cb8005586f8f0652d (127.0.98.2:36055): tablet 0969bb08f68145c9a501e5b65ba2af84 (table test-table [id=087ff7eff70642d0873028a07314a0bd]) successfully deleted
I20260925 13:17:53.773885 8414 tablet_bootstrap.cc:492] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160: No bootstrap required, opened a new log
I20260925 13:17:53.773936 8414 ts_tablet_manager.cc:1397] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.000s
I20260925 13:17:53.774096 8414 raft_consensus.cc:348] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:53.774158 8414 raft_consensus.cc:374] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:53.774174 8414 raft_consensus.cc:729] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b4e68537f845c0ba85219786abd160, State: Initialized, Role: FOLLOWER
I20260925 13:17:53.774214 8414 consensus_queue.cc:260] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:53.774297 8414 ts_tablet_manager.cc:1428] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260925 13:17:53.829676 8106 tablet.cc:2369] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:17:53.885183 8386 tablet.cc:2369] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:17:53.925455 8423 raft_consensus.cc:482] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:53.925585 8423 raft_consensus.cc:504] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:53.925726 8423 leader_election.cc:288] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69153cd48ea44695a251033a9e00cdeb (127.0.98.1:33769), 76b4e68537f845c0ba85219786abd160 (127.0.98.3:37949)
I20260925 13:17:53.926038 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42e09b79edb4c79909539ed77c679d8" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69153cd48ea44695a251033a9e00cdeb" is_pre_election: true
I20260925 13:17:53.926036 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42e09b79edb4c79909539ed77c679d8" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b4e68537f845c0ba85219786abd160" is_pre_election: true
I20260925 13:17:53.926139 8059 raft_consensus.cc:2454] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 0.
I20260925 13:17:53.926139 8339 raft_consensus.cc:2454] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 0.
I20260925 13:17:53.926307 8132 leader_election.cc:302] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69153cd48ea44695a251033a9e00cdeb, e5e6a16415d5473cb8005586f8f0652d; no voters:
I20260925 13:17:53.926397 8423 raft_consensus.cc:2781] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:53.926447 8423 raft_consensus.cc:482] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:53.926468 8423 raft_consensus.cc:3037] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:53.927136 8423 raft_consensus.cc:504] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:53.927249 8423 leader_election.cc:288] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 election: Requested vote from peers 69153cd48ea44695a251033a9e00cdeb (127.0.98.1:33769), 76b4e68537f845c0ba85219786abd160 (127.0.98.3:37949)
I20260925 13:17:53.927443 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42e09b79edb4c79909539ed77c679d8" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69153cd48ea44695a251033a9e00cdeb"
I20260925 13:17:53.927515 8059 raft_consensus.cc:3037] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:53.927459 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42e09b79edb4c79909539ed77c679d8" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b4e68537f845c0ba85219786abd160"
I20260925 13:17:53.927546 8339 raft_consensus.cc:3037] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:53.928126 8059 raft_consensus.cc:2454] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 1.
I20260925 13:17:53.928164 8339 raft_consensus.cc:2454] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 1.
I20260925 13:17:53.928316 8132 leader_election.cc:302] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b4e68537f845c0ba85219786abd160, e5e6a16415d5473cb8005586f8f0652d; no voters:
I20260925 13:17:53.928416 8423 raft_consensus.cc:2781] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:53.928475 8423 raft_consensus.cc:686] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [term 1 LEADER]: Becoming Leader. State: Replica: e5e6a16415d5473cb8005586f8f0652d, State: Running, Role: LEADER
I20260925 13:17:53.928529 8423 consensus_queue.cc:237] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:53.929051 7901 catalog_manager.cc:5495] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d reported cstate change: term changed from 0 to 1, leader changed from <none> to e5e6a16415d5473cb8005586f8f0652d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e5e6a16415d5473cb8005586f8f0652d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } health_report { overall_health: UNKNOWN } } }
Sep 25 13:17:53 dist-test-slave-t8d9 krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:17:53.976496 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.970239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57868 (local address 127.0.98.62:39083)
0925 13:17:53.970321 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:53.970323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:53.970390 (+ 67us) server_negotiation.cc:407] Connection header received
0925 13:17:53.970444 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:53.970445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:53.970453 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:53.970499 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:53.970895 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.971382 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.972070 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.972237 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.972278 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.974383 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:53.974384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:53.974384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:53.974390 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:53.975512 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.975824 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.975825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.975826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.975871 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:53.976018 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:53.976018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:53.976019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:53.976118 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:53.976170 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:53.976302 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 13:17:53.976381 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":58}
I20260925 13:17:53.976563 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:53.970190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39083 (local address 127.0.0.1:57868)
0925 13:17:53.970323 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:53.970333 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:53.970398 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:53.970510 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:53.970510 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:53.970796 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:53.970845 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.970847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.971463 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.971464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.972005 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:53.972006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:53.972308 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:53.972309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:53.972356 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:53.972871 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:53.972875 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:53.974303 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:53.975585 (+ 1282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:53.975592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:53.975595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:53.975764 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:53.975924 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:53.975925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:53.975925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:53.975973 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:53.976168 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:53.976176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:53.976220 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 13:17:53.976275 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 13:17:53.976445 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":69,"thread_start_us":68,"threads_started":1}
I20260925 13:17:53.978397 7901 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57868:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 13:17:53.978581 7901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:53.979925 8039 tablet_service.cc:1467] Processing CreateTablet for tablet a196826254244e2686006f48030065ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92f87aab2fc142a18e725d6f8682251d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:17:53.979972 8319 tablet_service.cc:1467] Processing CreateTablet for tablet a196826254244e2686006f48030065ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92f87aab2fc142a18e725d6f8682251d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:17:53.980057 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a196826254244e2686006f48030065ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:53.980114 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a196826254244e2686006f48030065ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:53.981132 8414 tablet_bootstrap.cc:492] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160: Bootstrap starting.
I20260925 13:17:53.981266 8179 tablet_service.cc:1467] Processing CreateTablet for tablet a196826254244e2686006f48030065ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92f87aab2fc142a18e725d6f8682251d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 13:17:53.981376 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a196826254244e2686006f48030065ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:53.981505 8414 tablet_bootstrap.cc:654] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:53.981837 8413 tablet_bootstrap.cc:492] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb: Bootstrap starting.
I20260925 13:17:53.982165 8414 tablet_bootstrap.cc:492] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160: No bootstrap required, opened a new log
I20260925 13:17:53.982175 8413 tablet_bootstrap.cc:654] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:53.982206 8414 ts_tablet_manager.cc:1397] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:53.982338 8414 raft_consensus.cc:348] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:53.982411 8414 raft_consensus.cc:374] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:53.982431 8414 raft_consensus.cc:729] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b4e68537f845c0ba85219786abd160, State: Initialized, Role: FOLLOWER
I20260925 13:17:53.982472 8414 consensus_queue.cc:260] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:53.982553 8414 ts_tablet_manager.cc:1428] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:53.982669 8416 tablet_bootstrap.cc:492] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d: Bootstrap starting.
I20260925 13:17:53.982717 8413 tablet_bootstrap.cc:492] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb: No bootstrap required, opened a new log
I20260925 13:17:53.982697 8421 tablet_replica.cc:439] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } }
I20260925 13:17:53.982759 8413 ts_tablet_manager.cc:1397] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 13:17:53.982787 8421 tablet_replica.cc:442] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:53.982885 8413 raft_consensus.cc:348] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:53.982955 8413 raft_consensus.cc:374] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:53.982975 8413 raft_consensus.cc:729] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69153cd48ea44695a251033a9e00cdeb, State: Initialized, Role: FOLLOWER
I20260925 13:17:53.982992 8416 tablet_bootstrap.cc:654] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:53.983011 8413 consensus_queue.cc:260] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:53.983103 8413 ts_tablet_manager.cc:1428] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:53.983140 8419 tablet_replica.cc:439] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } }
I20260925 13:17:53.983197 8419 tablet_replica.cc:442] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:53.983500 8416 tablet_bootstrap.cc:492] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d: No bootstrap required, opened a new log
I20260925 13:17:53.983558 8416 ts_tablet_manager.cc:1397] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:53.983675 8416 raft_consensus.cc:348] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:53.983736 8416 raft_consensus.cc:374] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:53.983753 8416 raft_consensus.cc:729] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e6a16415d5473cb8005586f8f0652d, State: Initialized, Role: FOLLOWER
I20260925 13:17:53.983798 8416 consensus_queue.cc:260] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:53.983886 8416 ts_tablet_manager.cc:1428] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:53.983901 8423 tablet_replica.cc:439] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } }
I20260925 13:17:53.983964 8423 tablet_replica.cc:442] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:54.118604 8423 raft_consensus.cc:482] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:54.118741 8423 raft_consensus.cc:504] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.118882 8423 leader_election.cc:288] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69153cd48ea44695a251033a9e00cdeb (127.0.98.1:33769), 76b4e68537f845c0ba85219786abd160 (127.0.98.3:37949)
I20260925 13:17:54.119128 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a196826254244e2686006f48030065ef" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69153cd48ea44695a251033a9e00cdeb" is_pre_election: true
I20260925 13:17:54.119135 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a196826254244e2686006f48030065ef" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b4e68537f845c0ba85219786abd160" is_pre_election: true
I20260925 13:17:54.119226 8339 raft_consensus.cc:2454] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 0.
I20260925 13:17:54.119226 8059 raft_consensus.cc:2454] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 0.
I20260925 13:17:54.119381 8132 leader_election.cc:302] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b4e68537f845c0ba85219786abd160, e5e6a16415d5473cb8005586f8f0652d; no voters:
I20260925 13:17:54.119469 8423 raft_consensus.cc:2781] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:54.119544 8423 raft_consensus.cc:482] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:54.119567 8423 raft_consensus.cc:3037] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:54.120267 8423 raft_consensus.cc:504] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.120404 8423 leader_election.cc:288] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 election: Requested vote from peers 69153cd48ea44695a251033a9e00cdeb (127.0.98.1:33769), 76b4e68537f845c0ba85219786abd160 (127.0.98.3:37949)
I20260925 13:17:54.120565 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a196826254244e2686006f48030065ef" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69153cd48ea44695a251033a9e00cdeb"
I20260925 13:17:54.120580 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a196826254244e2686006f48030065ef" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b4e68537f845c0ba85219786abd160"
I20260925 13:17:54.120648 8059 raft_consensus.cc:3037] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:54.120648 8339 raft_consensus.cc:3037] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:54.121188 8339 raft_consensus.cc:2454] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 1.
I20260925 13:17:54.121248 8059 raft_consensus.cc:2454] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 1.
I20260925 13:17:54.121423 8132 leader_election.cc:302] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69153cd48ea44695a251033a9e00cdeb, e5e6a16415d5473cb8005586f8f0652d; no voters:
I20260925 13:17:54.121515 8423 raft_consensus.cc:2781] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:54.121608 8423 raft_consensus.cc:686] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [term 1 LEADER]: Becoming Leader. State: Replica: e5e6a16415d5473cb8005586f8f0652d, State: Running, Role: LEADER
I20260925 13:17:54.121661 8423 consensus_queue.cc:237] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.121968 8420 tablet_replica.cc:439] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d: TxnStatusTablet state changed. Reason: New leader e5e6a16415d5473cb8005586f8f0652d. Latest consensus state: current_term: 1 leader_uuid: "e5e6a16415d5473cb8005586f8f0652d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } }
I20260925 13:17:54.122043 8420 tablet_replica.cc:442] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:17:54.122179 8452 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 13:17:54.122298 8452 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 13:17:54.122320 7901 catalog_manager.cc:5495] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d reported cstate change: term changed from 0 to 1, leader changed from <none> to e5e6a16415d5473cb8005586f8f0652d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e5e6a16415d5473cb8005586f8f0652d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } health_report { overall_health: UNKNOWN } } }
I20260925 13:17:54.131834 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:54.129216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.0.98.2:36055)
0925 13:17:54.129269 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:54.129272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:54.129290 (+ 18us) server_negotiation.cc:407] Connection header received
0925 13:17:54.129366 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:54.129367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:54.129378 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:54.129434 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:54.129551 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.130293 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.131008 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.131308 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.131371 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:54.131517 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:54.131550 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:54.131614 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:54.131732 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 13:17:54.131753 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:17:54.131839 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:54.129136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36055 (local address 127.0.0.1:51900)
0925 13:17:54.129234 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:54.129252 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:54.129311 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:54.129444 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:54.129444 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:54.129445 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:17:54.129491 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:54.129493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.130362 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.130363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:54.130939 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:54.130941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.131368 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.131368 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:54.131424 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:54.131444 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:17:54.131658 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:17:54.131671 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 13:17:54.131725 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 13:17:54.131776 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 13:17:54.132870 8339 raft_consensus.cc:1261] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Refusing update from remote peer e5e6a16415d5473cb8005586f8f0652d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:54.132903 8059 raft_consensus.cc:1261] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Refusing update from remote peer e5e6a16415d5473cb8005586f8f0652d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:54.133133 8420 consensus_queue.cc:1035] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:54.133273 8423 consensus_queue.cc:1035] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:54.133790 8419 tablet_replica.cc:439] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb: TxnStatusTablet state changed. Reason: New leader e5e6a16415d5473cb8005586f8f0652d. Latest consensus state: current_term: 1 leader_uuid: "e5e6a16415d5473cb8005586f8f0652d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } }
I20260925 13:17:54.133893 8419 tablet_replica.cc:442] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:54.134291 8421 tablet_replica.cc:439] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160: TxnStatusTablet state changed. Reason: New leader e5e6a16415d5473cb8005586f8f0652d. Latest consensus state: current_term: 1 leader_uuid: "e5e6a16415d5473cb8005586f8f0652d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } }
I20260925 13:17:54.134404 8421 tablet_replica.cc:442] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:54.134738 8420 tablet_replica.cc:439] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5e6a16415d5473cb8005586f8f0652d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } }
I20260925 13:17:54.134826 8420 tablet_replica.cc:442] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:17:54.135054 8426 mvcc.cc:205] Tried to move back new op lower bound from 7333241954846613504 to 7333241954802761728. Current Snapshot: MvccSnapshot[applied={T|T < 7333241954846613504 or (T in {7333241954846613504})}]
I20260925 13:17:54.135417 8423 tablet_replica.cc:439] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5e6a16415d5473cb8005586f8f0652d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } }
I20260925 13:17:54.135496 8423 tablet_replica.cc:442] T a196826254244e2686006f48030065ef P e5e6a16415d5473cb8005586f8f0652d: This TxnStatusTablet replica's current role is: LEADER
I20260925 13:17:54.135783 8421 tablet_replica.cc:439] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5e6a16415d5473cb8005586f8f0652d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } }
I20260925 13:17:54.135866 8421 tablet_replica.cc:442] T a196826254244e2686006f48030065ef P 76b4e68537f845c0ba85219786abd160: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:54.135839 8428 mvcc.cc:205] Tried to move back new op lower bound from 7333241954846613504 to 7333241954802761728. Current Snapshot: MvccSnapshot[applied={T|T < 7333241954846613504 or (T in {7333241954846613504})}]
I20260925 13:17:54.136348 8419 tablet_replica.cc:439] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5e6a16415d5473cb8005586f8f0652d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } } }
I20260925 13:17:54.136438 8419 tablet_replica.cc:442] T a196826254244e2686006f48030065ef P 69153cd48ea44695a251033a9e00cdeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 13:17:54.141852 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:54.139096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47009 (local address 127.0.98.2:36055)
0925 13:17:54.139135 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:54.139136 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:54.139146 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:17:54.139194 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:54.139194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:54.139205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:54.139274 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:54.139377 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.140101 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.141189 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.141367 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.141429 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:54.141611 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:54.141650 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:54.141690 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:54.141751 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 13:17:54.141770 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:17:54.141978 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:54.138905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36055 (local address 127.0.98.2:47009)
0925 13:17:54.139076 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:54.139089 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:54.139141 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:54.139278 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:54.139279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:54.139281 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 13:17:54.139330 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:54.139331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.140191 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.140191 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:54.141106 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:54.141109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.141436 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.141436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:54.141490 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:54.141507 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 13:17:54.141700 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 13:17:54.141702 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:17:54.141747 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 13:17:54.141794 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260925 13:17:54.144140 8059 raft_consensus.cc:1261] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Refusing update from remote peer e5e6a16415d5473cb8005586f8f0652d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:54.144153 8339 raft_consensus.cc:1261] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Refusing update from remote peer e5e6a16415d5473cb8005586f8f0652d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:54.144335 8460 consensus_queue.cc:1035] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:54.144479 8420 consensus_queue.cc:1035] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:54.161727 7901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:54.161835 7901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:54.162495 7901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329: Sending DeleteTablet for 3 replicas of tablet a42e09b79edb4c79909539ed77c679d8
I20260925 13:17:54.162743 8039 tablet_service.cc:1514] Processing DeleteTablet for tablet a42e09b79edb4c79909539ed77c679d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47888
I20260925 13:17:54.162763 8319 tablet_service.cc:1514] Processing DeleteTablet for tablet a42e09b79edb4c79909539ed77c679d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57082
I20260925 13:17:54.162839 8436 tablet_replica.cc:330] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb: stopping tablet replica
I20260925 13:17:54.162889 8436 raft_consensus.cc:2229] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:54.162918 8436 raft_consensus.cc:2258] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:54.162981 8435 tablet_replica.cc:330] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160: stopping tablet replica
I20260925 13:17:54.163028 8435 raft_consensus.cc:2229] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:54.163064 8435 raft_consensus.cc:2258] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:54.163197 8178 tablet_service.cc:1514] Processing DeleteTablet for tablet a42e09b79edb4c79909539ed77c679d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51886
I20260925 13:17:54.163242 8436 ts_tablet_manager.cc:1905] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:54.164492 8437 tablet_replica.cc:330] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d: stopping tablet replica
I20260925 13:17:54.164559 8437 raft_consensus.cc:2229] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:17:54.164629 8437 raft_consensus.cc:2258] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:54.164960 8437 ts_tablet_manager.cc:1905] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:54.166219 8436 ts_tablet_manager.cc:1918] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:17:54.166261 8437 ts_tablet_manager.cc:1918] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:17:54.166285 8436 log.cc:1196] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/wal/wals/a42e09b79edb4c79909539ed77c679d8
I20260925 13:17:54.166301 8437 log.cc:1196] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/a42e09b79edb4c79909539ed77c679d8
I20260925 13:17:54.166477 8437 ts_tablet_manager.cc:1939] T a42e09b79edb4c79909539ed77c679d8 P e5e6a16415d5473cb8005586f8f0652d: Deleting consensus metadata
I20260925 13:17:54.166477 8436 ts_tablet_manager.cc:1939] T a42e09b79edb4c79909539ed77c679d8 P 69153cd48ea44695a251033a9e00cdeb: Deleting consensus metadata
I20260925 13:17:54.166800 7888 catalog_manager.cc:4841] TS 69153cd48ea44695a251033a9e00cdeb (127.0.98.1:33769): tablet a42e09b79edb4c79909539ed77c679d8 (table test-table [id=7092ae5633c643bd8a0ce861377f41bc]) successfully deleted
I20260925 13:17:54.166855 7887 catalog_manager.cc:4841] TS e5e6a16415d5473cb8005586f8f0652d (127.0.98.2:36055): tablet a42e09b79edb4c79909539ed77c679d8 (table test-table [id=7092ae5633c643bd8a0ce861377f41bc]) successfully deleted
I20260925 13:17:54.167037 8435 ts_tablet_manager.cc:1905] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:54.168159 8435 ts_tablet_manager.cc:1918] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:17:54.168205 8435 log.cc:1196] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/a42e09b79edb4c79909539ed77c679d8
I20260925 13:17:54.168365 8435 ts_tablet_manager.cc:1939] T a42e09b79edb4c79909539ed77c679d8 P 76b4e68537f845c0ba85219786abd160: Deleting consensus metadata
I20260925 13:17:54.168633 7888 catalog_manager.cc:4841] TS 76b4e68537f845c0ba85219786abd160 (127.0.98.3:37949): tablet a42e09b79edb4c79909539ed77c679d8 (table test-table [id=7092ae5633c643bd8a0ce861377f41bc]) successfully deleted
Sep 25 13:17:54 dist-test-slave-t8d9 krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 13:17:54 dist-test-slave-t8d9 krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:54 dist-test-slave-t8d9 krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 13:17:54.189227 7971 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:54.183079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57872 (local address 127.0.98.62:39083)
0925 13:17:54.183147 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:54.183149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:54.183255 (+ 106us) server_negotiation.cc:407] Connection header received
0925 13:17:54.183328 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:54.183329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:54.183338 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:54.183384 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:54.184046 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.184581 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.185713 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.185900 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.185970 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:54.189157 (+ 3187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57872: BlockingRecv error: recv got EOF from 127.0.0.1:57872 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 13:17:54 dist-test-slave-t8d9 krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 13:17:54.197043 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:54.190049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57888 (local address 127.0.98.62:39083)
0925 13:17:54.190092 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:54.190094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:54.190106 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:17:54.190169 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:54.190170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:54.190180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:54.190226 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:54.190658 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.191219 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.191902 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.192155 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.192210 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:54.194627 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:54.194628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:54.194629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:54.194636 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:54.195991 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:54.196363 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:54.196364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:54.196364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:54.196410 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:54.196582 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:54.196583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:54.196583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:54.196680 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:54.196742 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:54.196875 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 13:17:54.196975 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:17:54.198544 7901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:17:54.198695 7901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:54.199987 8319 tablet_service.cc:1467] Processing CreateTablet for tablet 8266e11d6a2344538da950b00ac2e09a (DEFAULT_TABLE table=test-table [id=d003d89827724af595dd2aa2f67d5d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:54.199986 8039 tablet_service.cc:1467] Processing CreateTablet for tablet 8266e11d6a2344538da950b00ac2e09a (DEFAULT_TABLE table=test-table [id=d003d89827724af595dd2aa2f67d5d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:54.200011 8178 tablet_service.cc:1467] Processing CreateTablet for tablet 8266e11d6a2344538da950b00ac2e09a (DEFAULT_TABLE table=test-table [id=d003d89827724af595dd2aa2f67d5d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:54.200114 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8266e11d6a2344538da950b00ac2e09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:54.200114 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8266e11d6a2344538da950b00ac2e09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:54.200145 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8266e11d6a2344538da950b00ac2e09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:54.201347 8413 tablet_bootstrap.cc:492] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb: Bootstrap starting.
I20260925 13:17:54.201370 8414 tablet_bootstrap.cc:492] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160: Bootstrap starting.
I20260925 13:17:54.201587 8416 tablet_bootstrap.cc:492] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d: Bootstrap starting.
I20260925 13:17:54.201776 8414 tablet_bootstrap.cc:654] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:54.201853 8413 tablet_bootstrap.cc:654] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:54.201881 8416 tablet_bootstrap.cc:654] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:54.202474 8414 tablet_bootstrap.cc:492] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160: No bootstrap required, opened a new log
I20260925 13:17:54.202520 8416 tablet_bootstrap.cc:492] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d: No bootstrap required, opened a new log
I20260925 13:17:54.202527 8414 ts_tablet_manager.cc:1397] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:54.202569 8416 ts_tablet_manager.cc:1397] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 13:17:54.202677 8414 raft_consensus.cc:348] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.202694 8416 raft_consensus.cc:348] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.202746 8414 raft_consensus.cc:374] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:54.202759 8416 raft_consensus.cc:374] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:54.202766 8414 raft_consensus.cc:729] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b4e68537f845c0ba85219786abd160, State: Initialized, Role: FOLLOWER
I20260925 13:17:54.202778 8416 raft_consensus.cc:729] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e6a16415d5473cb8005586f8f0652d, State: Initialized, Role: FOLLOWER
I20260925 13:17:54.202806 8413 tablet_bootstrap.cc:492] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb: No bootstrap required, opened a new log
I20260925 13:17:54.202802 8414 consensus_queue.cc:260] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.202821 8416 consensus_queue.cc:260] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.202840 8413 ts_tablet_manager.cc:1397] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 13:17:54.202888 8414 ts_tablet_manager.cc:1428] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:54.202903 8416 ts_tablet_manager.cc:1428] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:54.202993 8413 raft_consensus.cc:348] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.203053 8413 raft_consensus.cc:374] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:54.203070 8413 raft_consensus.cc:729] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69153cd48ea44695a251033a9e00cdeb, State: Initialized, Role: FOLLOWER
I20260925 13:17:54.203105 8413 consensus_queue.cc:260] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.203187 8413 ts_tablet_manager.cc:1428] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:54.322922 8421 raft_consensus.cc:482] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:54.323015 8421 raft_consensus.cc:504] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.323355 8421 leader_election.cc:288] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69153cd48ea44695a251033a9e00cdeb (127.0.98.1:33769), e5e6a16415d5473cb8005586f8f0652d (127.0.98.2:36055)
I20260925 13:17:54.326227 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:54.323600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47033 (local address 127.0.98.2:36055)
0925 13:17:54.323671 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:54.323673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:54.323739 (+ 66us) server_negotiation.cc:407] Connection header received
0925 13:17:54.323797 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:54.323797 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:54.323810 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:54.323857 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:54.324018 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.324491 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.325475 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.325932 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.325992 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:54.326005 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:54.326123 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 13:17:54.326161 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 13:17:54.326278 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:54.323649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60777 (local address 127.0.98.1:33769)
0925 13:17:54.323746 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:54.323748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:54.323773 (+ 25us) server_negotiation.cc:407] Connection header received
0925 13:17:54.323798 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:54.323799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:54.323809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:54.323856 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:17:54.324006 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.324485 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.325482 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.325956 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.326015 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:54.326026 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:54.326166 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 13:17:54.326200 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":49,"threads_started":1}
I20260925 13:17:54.326447 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:54.323540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33769 (local address 127.0.98.3:60777)
0925 13:17:54.323677 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:54.323688 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:54.323741 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:54.323867 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:54.323868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:54.323869 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:54.323942 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:54.323944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.324571 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.324572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:54.325392 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:54.325394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.326042 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.326043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:54.326106 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:54.326109 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:17:54.326153 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 13:17:54.326193 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":56,"thread_start_us":62,"threads_started":1}
I20260925 13:17:54.326575 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:54.323533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36055 (local address 127.0.98.3:47033)
0925 13:17:54.323669 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:54.323679 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:54.323745 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:54.323870 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:54.323871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:54.323873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:17:54.323956 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:54.323957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.324563 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.324563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:54.325389 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:54.325391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.325998 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.325998 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:54.326065 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:54.326069 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:17:54.326112 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:17:54.326150 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":34,"thread_start_us":72,"threads_started":1}
I20260925 13:17:54.326735 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8266e11d6a2344538da950b00ac2e09a" candidate_uuid: "76b4e68537f845c0ba85219786abd160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69153cd48ea44695a251033a9e00cdeb" is_pre_election: true
I20260925 13:17:54.326828 8059 raft_consensus.cc:2454] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76b4e68537f845c0ba85219786abd160 in term 0.
I20260925 13:17:54.326865 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8266e11d6a2344538da950b00ac2e09a" candidate_uuid: "76b4e68537f845c0ba85219786abd160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e6a16415d5473cb8005586f8f0652d" is_pre_election: true
I20260925 13:17:54.326989 8199 raft_consensus.cc:2454] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76b4e68537f845c0ba85219786abd160 in term 0.
I20260925 13:17:54.326984 8272 leader_election.cc:302] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69153cd48ea44695a251033a9e00cdeb, 76b4e68537f845c0ba85219786abd160; no voters:
I20260925 13:17:54.327118 8421 raft_consensus.cc:2781] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:54.327150 8421 raft_consensus.cc:482] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:54.327162 8421 raft_consensus.cc:3037] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:54.327772 8421 raft_consensus.cc:504] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.327883 8421 leader_election.cc:288] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [CANDIDATE]: Term 1 election: Requested vote from peers 69153cd48ea44695a251033a9e00cdeb (127.0.98.1:33769), e5e6a16415d5473cb8005586f8f0652d (127.0.98.2:36055)
I20260925 13:17:54.328025 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8266e11d6a2344538da950b00ac2e09a" candidate_uuid: "76b4e68537f845c0ba85219786abd160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69153cd48ea44695a251033a9e00cdeb"
I20260925 13:17:54.328094 8059 raft_consensus.cc:3037] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:54.328058 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8266e11d6a2344538da950b00ac2e09a" candidate_uuid: "76b4e68537f845c0ba85219786abd160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e6a16415d5473cb8005586f8f0652d"
I20260925 13:17:54.328133 8199 raft_consensus.cc:3037] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:54.328715 8059 raft_consensus.cc:2454] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76b4e68537f845c0ba85219786abd160 in term 1.
I20260925 13:17:54.328858 8272 leader_election.cc:302] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69153cd48ea44695a251033a9e00cdeb, 76b4e68537f845c0ba85219786abd160; no voters:
I20260925 13:17:54.328936 8421 raft_consensus.cc:2781] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:54.328967 8199 raft_consensus.cc:2454] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76b4e68537f845c0ba85219786abd160 in term 1.
I20260925 13:17:54.329028 8421 raft_consensus.cc:686] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [term 1 LEADER]: Becoming Leader. State: Replica: 76b4e68537f845c0ba85219786abd160, State: Running, Role: LEADER
I20260925 13:17:54.329103 8421 consensus_queue.cc:237] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.329841 7901 catalog_manager.cc:5495] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76b4e68537f845c0ba85219786abd160 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "76b4e68537f845c0ba85219786abd160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } health_report { overall_health: HEALTHY } } }
I20260925 13:17:54.352133 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:54.349891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.0.98.3:37949)
0925 13:17:54.350036 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:54.350038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:54.350047 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:54.350088 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:54.350089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:54.350108 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:54.350177 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:54.350266 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.350752 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.351502 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.351694 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.351736 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:54.351860 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:54.351963 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:54.352011 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:54.352045 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 13:17:54.352063 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260925 13:17:54.353042 8059 raft_consensus.cc:1261] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Refusing update from remote peer 76b4e68537f845c0ba85219786abd160: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:54.353188 8199 raft_consensus.cc:1261] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d [term 1 FOLLOWER]: Refusing update from remote peer 76b4e68537f845c0ba85219786abd160: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:54.353309 8482 consensus_queue.cc:1035] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:54.353425 8421 consensus_queue.cc:1035] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:54.356951 7900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:54.357048 7900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:54.357656 7900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329: Sending DeleteTablet for 3 replicas of tablet 8266e11d6a2344538da950b00ac2e09a
I20260925 13:17:54.357919 8178 tablet_service.cc:1514] Processing DeleteTablet for tablet 8266e11d6a2344538da950b00ac2e09a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51886
I20260925 13:17:54.357971 8319 tablet_service.cc:1514] Processing DeleteTablet for tablet 8266e11d6a2344538da950b00ac2e09a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57082
I20260925 13:17:54.358044 8437 tablet_replica.cc:330] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d: stopping tablet replica
I20260925 13:17:54.358073 8435 tablet_replica.cc:330] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160: stopping tablet replica
I20260925 13:17:54.358108 8437 raft_consensus.cc:2229] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:54.358126 8435 raft_consensus.cc:2229] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:17:54.358152 8437 raft_consensus.cc:2258] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:54.358206 8435 raft_consensus.cc:2258] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:54.358217 8039 tablet_service.cc:1514] Processing DeleteTablet for tablet 8266e11d6a2344538da950b00ac2e09a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47888
I20260925 13:17:54.358309 8436 tablet_replica.cc:330] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb: stopping tablet replica
I20260925 13:17:54.358361 8436 raft_consensus.cc:2229] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:54.358409 8436 raft_consensus.cc:2258] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:54.358656 8435 ts_tablet_manager.cc:1905] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:54.358641 7900 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:17:54.358798 7900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:17:54.359058 8436 ts_tablet_manager.cc:1905] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:54.359907 8178 tablet_service.cc:1467] Processing CreateTablet for tablet 512f3119d2bb404ebe86a5b4dfabe4c5 (DEFAULT_TABLE table=test-table [id=d58f09aca51f40e38e3beb7f29be22ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:54.360021 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512f3119d2bb404ebe86a5b4dfabe4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:54.360136 8435 ts_tablet_manager.cc:1918] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:54.360183 8435 log.cc:1196] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/8266e11d6a2344538da950b00ac2e09a
I20260925 13:17:54.360384 8435 ts_tablet_manager.cc:1939] T 8266e11d6a2344538da950b00ac2e09a P 76b4e68537f845c0ba85219786abd160: Deleting consensus metadata
I20260925 13:17:54.360445 8436 ts_tablet_manager.cc:1918] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:54.360495 8436 log.cc:1196] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-0/wal/wals/8266e11d6a2344538da950b00ac2e09a
I20260925 13:17:54.360661 8436 ts_tablet_manager.cc:1939] T 8266e11d6a2344538da950b00ac2e09a P 69153cd48ea44695a251033a9e00cdeb: Deleting consensus metadata
I20260925 13:17:54.360921 8437 ts_tablet_manager.cc:1905] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:54.361372 8416 tablet_bootstrap.cc:492] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d: Bootstrap starting.
I20260925 13:17:54.361639 7888 catalog_manager.cc:4841] TS 69153cd48ea44695a251033a9e00cdeb (127.0.98.1:33769): tablet 8266e11d6a2344538da950b00ac2e09a (table test-table [id=d003d89827724af595dd2aa2f67d5d38]) successfully deleted
I20260925 13:17:54.361696 7888 catalog_manager.cc:4841] TS 76b4e68537f845c0ba85219786abd160 (127.0.98.3:37949): tablet 8266e11d6a2344538da950b00ac2e09a (table test-table [id=d003d89827724af595dd2aa2f67d5d38]) successfully deleted
I20260925 13:17:54.361696 8416 tablet_bootstrap.cc:654] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:54.361820 8319 tablet_service.cc:1467] Processing CreateTablet for tablet 512f3119d2bb404ebe86a5b4dfabe4c5 (DEFAULT_TABLE table=test-table [id=d58f09aca51f40e38e3beb7f29be22ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:54.361908 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512f3119d2bb404ebe86a5b4dfabe4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:54.362171 8416 tablet_bootstrap.cc:492] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d: No bootstrap required, opened a new log
I20260925 13:17:54.362234 8416 ts_tablet_manager.cc:1397] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 13:17:54.362339 8416 raft_consensus.cc:348] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.362401 8416 raft_consensus.cc:374] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:54.362417 8416 raft_consensus.cc:729] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e6a16415d5473cb8005586f8f0652d, State: Initialized, Role: FOLLOWER
I20260925 13:17:54.362190 8437 ts_tablet_manager.cc:1918] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:17:54.362520 8437 log.cc:1196] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/8266e11d6a2344538da950b00ac2e09a
I20260925 13:17:54.362720 8437 ts_tablet_manager.cc:1939] T 8266e11d6a2344538da950b00ac2e09a P e5e6a16415d5473cb8005586f8f0652d: Deleting consensus metadata
I20260925 13:17:54.362450 8416 consensus_queue.cc:260] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.362931 8416 ts_tablet_manager.cc:1428] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 13:17:54.363087 8414 tablet_bootstrap.cc:492] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160: Bootstrap starting.
I20260925 13:17:54.363233 8039 tablet_service.cc:1467] Processing CreateTablet for tablet 512f3119d2bb404ebe86a5b4dfabe4c5 (DEFAULT_TABLE table=test-table [id=d58f09aca51f40e38e3beb7f29be22ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:17:54.363343 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512f3119d2bb404ebe86a5b4dfabe4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:54.363479 8414 tablet_bootstrap.cc:654] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:54.364012 8414 tablet_bootstrap.cc:492] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160: No bootstrap required, opened a new log
I20260925 13:17:54.364050 8414 ts_tablet_manager.cc:1397] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:54.364182 8414 raft_consensus.cc:348] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.364243 8414 raft_consensus.cc:374] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:54.364260 8414 raft_consensus.cc:729] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b4e68537f845c0ba85219786abd160, State: Initialized, Role: FOLLOWER
I20260925 13:17:54.364300 8414 consensus_queue.cc:260] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.364389 8414 ts_tablet_manager.cc:1428] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:54.364563 8413 tablet_bootstrap.cc:492] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb: Bootstrap starting.
I20260925 13:17:54.364686 7887 catalog_manager.cc:4841] TS e5e6a16415d5473cb8005586f8f0652d (127.0.98.2:36055): tablet 8266e11d6a2344538da950b00ac2e09a (table test-table [id=d003d89827724af595dd2aa2f67d5d38]) successfully deleted
I20260925 13:17:54.364917 8413 tablet_bootstrap.cc:654] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:54.365775 8413 tablet_bootstrap.cc:492] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb: No bootstrap required, opened a new log
I20260925 13:17:54.365818 8413 ts_tablet_manager.cc:1397] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 13:17:54.365947 8413 raft_consensus.cc:348] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.366009 8413 raft_consensus.cc:374] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:54.366027 8413 raft_consensus.cc:729] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69153cd48ea44695a251033a9e00cdeb, State: Initialized, Role: FOLLOWER
I20260925 13:17:54.366066 8413 consensus_queue.cc:260] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.366151 8413 ts_tablet_manager.cc:1428] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:54.446816 8460 raft_consensus.cc:482] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:17:54.446920 8460 raft_consensus.cc:504] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.447077 8460 leader_election.cc:288] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69153cd48ea44695a251033a9e00cdeb (127.0.98.1:33769), 76b4e68537f845c0ba85219786abd160 (127.0.98.3:37949)
I20260925 13:17:54.447310 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512f3119d2bb404ebe86a5b4dfabe4c5" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69153cd48ea44695a251033a9e00cdeb" is_pre_election: true
I20260925 13:17:54.447409 8059 raft_consensus.cc:2454] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 0.
I20260925 13:17:54.447376 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512f3119d2bb404ebe86a5b4dfabe4c5" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b4e68537f845c0ba85219786abd160" is_pre_election: true
I20260925 13:17:54.447499 8339 raft_consensus.cc:2454] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 0.
I20260925 13:17:54.447598 8132 leader_election.cc:302] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69153cd48ea44695a251033a9e00cdeb, e5e6a16415d5473cb8005586f8f0652d; no voters:
I20260925 13:17:54.447708 8460 raft_consensus.cc:2781] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:17:54.447765 8460 raft_consensus.cc:482] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:17:54.447785 8460 raft_consensus.cc:3037] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:54.448747 8460 raft_consensus.cc:504] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.448884 8460 leader_election.cc:288] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 election: Requested vote from peers 69153cd48ea44695a251033a9e00cdeb (127.0.98.1:33769), 76b4e68537f845c0ba85219786abd160 (127.0.98.3:37949)
I20260925 13:17:54.449074 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512f3119d2bb404ebe86a5b4dfabe4c5" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69153cd48ea44695a251033a9e00cdeb"
I20260925 13:17:54.449148 8059 raft_consensus.cc:3037] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:54.449793 8059 raft_consensus.cc:2454] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 1.
I20260925 13:17:54.449905 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512f3119d2bb404ebe86a5b4dfabe4c5" candidate_uuid: "e5e6a16415d5473cb8005586f8f0652d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b4e68537f845c0ba85219786abd160"
I20260925 13:17:54.449966 8132 leader_election.cc:302] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69153cd48ea44695a251033a9e00cdeb, e5e6a16415d5473cb8005586f8f0652d; no voters:
I20260925 13:17:54.449985 8339 raft_consensus.cc:3037] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:54.450042 8460 raft_consensus.cc:2781] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:54.450080 8460 raft_consensus.cc:686] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [term 1 LEADER]: Becoming Leader. State: Replica: e5e6a16415d5473cb8005586f8f0652d, State: Running, Role: LEADER
I20260925 13:17:54.450131 8460 consensus_queue.cc:237] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } }
I20260925 13:17:54.450670 7900 catalog_manager.cc:5495] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d reported cstate change: term changed from 0 to 1, leader changed from <none> to e5e6a16415d5473cb8005586f8f0652d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e5e6a16415d5473cb8005586f8f0652d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e6a16415d5473cb8005586f8f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 } health_report { overall_health: UNKNOWN } } }
I20260925 13:17:54.450902 8339 raft_consensus.cc:2454] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e6a16415d5473cb8005586f8f0652d in term 1.
I20260925 13:17:54.463253 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:54.461043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.0.98.2:36055)
0925 13:17:54.461081 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:54.461083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:54.461126 (+ 43us) server_negotiation.cc:407] Connection header received
0925 13:17:54.461186 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:54.461187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:54.461196 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:54.461246 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:17:54.461365 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.461898 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.462635 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:54.462852 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:54.462906 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:54.463033 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:17:54.463066 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:17:54.463107 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:54.463154 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 13:17:54.463172 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 13:17:54.465162 8339 raft_consensus.cc:1261] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Refusing update from remote peer e5e6a16415d5473cb8005586f8f0652d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:54.465193 8059 raft_consensus.cc:1261] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Refusing update from remote peer e5e6a16415d5473cb8005586f8f0652d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:17:54.465385 8423 consensus_queue.cc:1035] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b4e68537f845c0ba85219786abd160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:54.465512 8420 consensus_queue.cc:1035] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [LEADER]: Connected to new peer: Peer: permanent_uuid: "69153cd48ea44695a251033a9e00cdeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:17:54.475121 7901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:54.475214 7901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:17:54.475731 7901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdd886dfe16f424f9173a574b861b329: Sending DeleteTablet for 3 replicas of tablet 512f3119d2bb404ebe86a5b4dfabe4c5
I20260925 13:17:54.475934 8039 tablet_service.cc:1514] Processing DeleteTablet for tablet 512f3119d2bb404ebe86a5b4dfabe4c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47888
I20260925 13:17:54.475955 8319 tablet_service.cc:1514] Processing DeleteTablet for tablet 512f3119d2bb404ebe86a5b4dfabe4c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57082
I20260925 13:17:54.476037 8436 tablet_replica.cc:330] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb: stopping tablet replica
I20260925 13:17:54.476043 8178 tablet_service.cc:1514] Processing DeleteTablet for tablet 512f3119d2bb404ebe86a5b4dfabe4c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:17:54 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51886
I20260925 13:17:54.476087 8436 raft_consensus.cc:2229] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:54.476118 8436 raft_consensus.cc:2258] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:54.476107 8435 tablet_replica.cc:330] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160: stopping tablet replica
I20260925 13:17:54.476158 8435 raft_consensus.cc:2229] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:17:54.476184 8437 tablet_replica.cc:330] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d: stopping tablet replica
I20260925 13:17:54.476197 8435 raft_consensus.cc:2258] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:54.476243 8437 raft_consensus.cc:2229] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:17:54.476301 8437 raft_consensus.cc:2258] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:17:54.476467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 7974
I20260925 13:17:54.476567 8435 ts_tablet_manager.cc:1905] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:54.476627 8437 ts_tablet_manager.cc:1905] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:54.476658 8436 ts_tablet_manager.cc:1905] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 69153cd48ea44695a251033a9e00cdeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:17:54.478137 8435 ts_tablet_manager.cc:1918] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:17:54.478137 8437 ts_tablet_manager.cc:1918] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 13:17:54.478195 8437 log.cc:1196] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/512f3119d2bb404ebe86a5b4dfabe4c5
I20260925 13:17:54.478195 8435 log.cc:1196] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/512f3119d2bb404ebe86a5b4dfabe4c5
I20260925 13:17:54.478389 8435 ts_tablet_manager.cc:1939] T 512f3119d2bb404ebe86a5b4dfabe4c5 P 76b4e68537f845c0ba85219786abd160: Deleting consensus metadata
I20260925 13:17:54.478389 8437 ts_tablet_manager.cc:1939] T 512f3119d2bb404ebe86a5b4dfabe4c5 P e5e6a16415d5473cb8005586f8f0652d: Deleting consensus metadata
I20260925 13:17:54.478679 7887 catalog_manager.cc:4841] TS e5e6a16415d5473cb8005586f8f0652d (127.0.98.2:36055): tablet 512f3119d2bb404ebe86a5b4dfabe4c5 (table test-table [id=d58f09aca51f40e38e3beb7f29be22ef]) successfully deleted
I20260925 13:17:54.482259 7888 catalog_manager.cc:4841] TS 76b4e68537f845c0ba85219786abd160 (127.0.98.3:37949): tablet 512f3119d2bb404ebe86a5b4dfabe4c5 (table test-table [id=d58f09aca51f40e38e3beb7f29be22ef]) successfully deleted
W20260925 13:17:54.482410 7888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33769 (error 108)
I20260925 13:17:54.482630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 8114
I20260925 13:17:54.483340 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:54.482701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33769 (local address 127.0.0.1:47892)
0925 13:17:54.483258 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:54.483300 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":34,"threads_started":1}
W20260925 13:17:54.483422 7888 catalog_manager.cc:4584] TS 69153cd48ea44695a251033a9e00cdeb (127.0.98.1:33769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 512f3119d2bb404ebe86a5b4dfabe4c5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33769: connect: Connection refused (error 111)
I20260925 13:17:54.487790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 8254
I20260925 13:17:54.492300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 7870
2026-09-25T13:17:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2548 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:17:54 dist-test-slave-t8d9 krb5kdc[8499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:17:54 dist-test-slave-t8d9 krb5kdc[8499](info): set up 2 sockets
Sep 25 13:17:54 dist-test-slave-t8d9 krb5kdc[8499](info): commencing operation
krb5kdc: starting...
W20260925 13:17:55.441872 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:17:55 dist-test-slave-t8d9 krb5kdc[8499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:17:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:17:55.553325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43437
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:55.632663 8512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:55.632915 8512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:55.632938 8512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:55.634150 8512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:55.634199 8512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:55.634212 8512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:55.634224 8512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:55.634234 8512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:55.635782 8512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:55.636003 8512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:55.636184 8512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:55.640854 8517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:55.640872 8520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:55.640934 8518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:55.641045 8512 server_base.cc:1034] running on GCE node
I20260925 13:17:55.641294 8512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:55.641613 8512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:55.642767 8512 hybrid_clock.cc:648] HybridClock initialized: now 1790342275642755 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T13:17:55Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1158 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:17:55 dist-test-slave-t8d9 krb5kdc[8525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:17:55 dist-test-slave-t8d9 krb5kdc[8525](info): set up 2 sockets
Sep 25 13:17:55 dist-test-slave-t8d9 krb5kdc[8525](info): commencing operation
krb5kdc: starting...
W20260925 13:17:56.597803 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:17:56 dist-test-slave-t8d9 krb5kdc[8525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:17:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:17:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:17:56.707692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:56.786201 8538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:56.786423 8538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:56.786455 8538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:56.787648 8538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:56.787689 8538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:56.787707 8538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:56.787722 8538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:56.787737 8538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:56.789239 8538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:56.789487 8538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:56.789717 8538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:56.794243 8546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:56.794261 8544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:56.794243 8543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:56.794831 8538 server_base.cc:1034] running on GCE node
I20260925 13:17:56.795056 8538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:56.795317 8538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:56.796465 8538 hybrid_clock.cc:648] HybridClock initialized: now 1790342276796450 us; error 27 us; skew 500 ppm
Sep 25 13:17:56 dist-test-slave-t8d9 krb5kdc[8525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:56.799152 8538 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:17:56.799578 8538 webserver.cc:460] Webserver started at http://127.0.98.62:33487/ using document root <none> and password file <none>
I20260925 13:17:56.799770 8538 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:56.799817 8538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:56.799937 8538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:56.800944 8538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "3cb2ca7bec12435e87973a1e957d9009"
format_stamp: "Formatted at 2026-09-25 13:17:56 on dist-test-slave-t8d9"
server_key: "d539f3de50c1b25904fb7217fea70441"
server_key_iv: "6d1ed79e76a32166b823b7aa9c515d95"
server_key_version: "encryptionkey@0"
I20260925 13:17:56.801277 8538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "3cb2ca7bec12435e87973a1e957d9009"
format_stamp: "Formatted at 2026-09-25 13:17:56 on dist-test-slave-t8d9"
server_key: "d539f3de50c1b25904fb7217fea70441"
server_key_iv: "6d1ed79e76a32166b823b7aa9c515d95"
server_key_version: "encryptionkey@0"
I20260925 13:17:56.802543 8538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 13:17:56.803223 8553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:56.803368 8538 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:56.803448 8538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "3cb2ca7bec12435e87973a1e957d9009"
format_stamp: "Formatted at 2026-09-25 13:17:56 on dist-test-slave-t8d9"
server_key: "d539f3de50c1b25904fb7217fea70441"
server_key_iv: "6d1ed79e76a32166b823b7aa9c515d95"
server_key_version: "encryptionkey@0"
I20260925 13:17:56.803514 8538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:56.811339 8538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:56.813405 8538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:56.813572 8538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:56.817222 8538 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41827
I20260925 13:17:56.817258 8605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41827 every 8 connection(s)
I20260925 13:17:56.817759 8538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:56.818349 8606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:17:56.820534 8606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009: Bootstrap starting.
I20260925 13:17:56.821105 8606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009: Neither blocks nor log segments found. Creating new log.
I20260925 13:17:56.821355 8606 log.cc:824] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:56.822008 8606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009: No bootstrap required, opened a new log
I20260925 13:17:56.822275 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 8538
I20260925 13:17:56.822358 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:17:56.822535 392 external_mini_cluster.cc:1428] Setting key ff13d9f47aeb98732ed1583dd48d2e6b
I20260925 13:17:56.824426 8606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb2ca7bec12435e87973a1e957d9009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } }
I20260925 13:17:56.824604 8606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:17:56.824638 8606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2ca7bec12435e87973a1e957d9009, State: Initialized, Role: FOLLOWER
I20260925 13:17:56.824790 8606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb2ca7bec12435e87973a1e957d9009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } }
I20260925 13:17:56.824863 8606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:56.824884 8606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:56.824918 8606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:17:56.825503 8606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb2ca7bec12435e87973a1e957d9009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } }
I20260925 13:17:56.825639 8606 leader_election.cc:302] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cb2ca7bec12435e87973a1e957d9009; no voters:
I20260925 13:17:56.825791 8606 leader_election.cc:288] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:17:56.825825 8611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:17:56.825999 8611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb2ca7bec12435e87973a1e957d9009, State: Running, Role: LEADER
I20260925 13:17:56.826030 8606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:56.826112 8611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb2ca7bec12435e87973a1e957d9009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } }
I20260925 13:17:56.826443 8611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cb2ca7bec12435e87973a1e957d9009. Latest consensus state: current_term: 1 leader_uuid: "3cb2ca7bec12435e87973a1e957d9009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb2ca7bec12435e87973a1e957d9009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } } }
I20260925 13:17:56.826514 8611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:56.826649 8612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cb2ca7bec12435e87973a1e957d9009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb2ca7bec12435e87973a1e957d9009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } } }
I20260925 13:17:56.826704 8612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:56.826805 8619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:56.827522 8619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:56.829104 8619 catalog_manager.cc:1347] Generated new cluster ID: c9bf13c48d0f42f69982bf49b24cdc5d
I20260925 13:17:56.829164 8619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:17:56 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:56.833928 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:56.822998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.0.98.62:41827)
0925 13:17:56.823185 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:56.823195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:56.823227 (+ 32us) server_negotiation.cc:407] Connection header received
0925 13:17:56.824395 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:56.824405 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:56.824478 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:56.824631 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:56.825297 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:56.826669 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:56.827734 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:56.828208 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:56.828310 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:56.830635 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:56.830639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:56.830639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:56.830650 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:56.832161 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:56.832469 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:56.832472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:56.832473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:56.832566 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:56.832742 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:56.832742 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:56.832743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:56.832975 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:56.833612 (+ 637us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:56.833673 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 13:17:56.833780 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260925 13:17:56.851328 8619 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:17:56.851877 8619 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:56.854466 8619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009: Generated new TSK 0
I20260925 13:17:56.854676 8619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:17:56.884392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41827
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:56.964803 8632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:56.965023 8632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:56.965055 8632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:56.966270 8632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:56.966312 8632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:56.966346 8632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:17:56.967834 8632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:56.968086 8632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:56.968293 8632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:56.972931 8640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:56.972946 8638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:56.972931 8637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:56.973176 8632 server_base.cc:1034] running on GCE node
I20260925 13:17:56.973434 8632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:56.973685 8632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:56.974814 8632 hybrid_clock.cc:648] HybridClock initialized: now 1790342276974806 us; error 26 us; skew 500 ppm
Sep 25 13:17:56 dist-test-slave-t8d9 krb5kdc[8525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:56.977180 8632 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:17:56.977669 8632 webserver.cc:460] Webserver started at http://127.0.98.1:37827/ using document root <none> and password file <none>
I20260925 13:17:56.977887 8632 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:56.977931 8632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:56.978057 8632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:56.979068 8632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "a2d69f0e8cc44e43ae4053f2e277d981"
format_stamp: "Formatted at 2026-09-25 13:17:56 on dist-test-slave-t8d9"
server_key: "4d3320c138bff8fb5a5e19ee5b57423a"
server_key_iv: "4cff05c239d07c5ffde97659d8d65b92"
server_key_version: "encryptionkey@0"
I20260925 13:17:56.979409 8632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a2d69f0e8cc44e43ae4053f2e277d981"
format_stamp: "Formatted at 2026-09-25 13:17:56 on dist-test-slave-t8d9"
server_key: "4d3320c138bff8fb5a5e19ee5b57423a"
server_key_iv: "4cff05c239d07c5ffde97659d8d65b92"
server_key_version: "encryptionkey@0"
I20260925 13:17:56.980676 8632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:17:56.981402 8647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:56.981565 8632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:56.981652 8632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "a2d69f0e8cc44e43ae4053f2e277d981"
format_stamp: "Formatted at 2026-09-25 13:17:56 on dist-test-slave-t8d9"
server_key: "4d3320c138bff8fb5a5e19ee5b57423a"
server_key_iv: "4cff05c239d07c5ffde97659d8d65b92"
server_key_version: "encryptionkey@0"
I20260925 13:17:56.981705 8632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:56.990834 8632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:56.992839 8632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:56.992961 8632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:56.993168 8632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:56.993448 8632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:56.993479 8632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:56.993499 8632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:56.993521 8632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:56.998589 8632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34095
I20260925 13:17:56.998648 8760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34095 every 8 connection(s)
I20260925 13:17:56.998926 8632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:17:56.999146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 8632
I20260925 13:17:56.999251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:17:56.999440 392 external_mini_cluster.cc:1428] Setting key 67190aeb1295d2d1707433c4717d6810
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 13:17:57.008297 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:56.999611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55563 (local address 127.0.98.62:41827)
0925 13:17:56.999659 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:56.999661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:56.999898 (+ 237us) server_negotiation.cc:407] Connection header received
0925 13:17:57.000874 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:57.000875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:57.000889 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:57.000956 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:57.001567 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.002280 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.003266 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.003583 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.003651 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.005801 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:57.005802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:57.005802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:57.005810 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:57.007200 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:57.007534 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:57.007534 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:57.007535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:57.007578 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:57.007783 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:57.007783 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:57.007783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:57.007885 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:57.007933 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:57.008160 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 13:17:57.008229 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:17:57.008435 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:56.999625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:55563)
0925 13:17:56.999777 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:56.999821 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:57.000766 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:57.001016 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:57.001020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:57.001274 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:57.001461 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.001465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.002357 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.002358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.003169 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.003173 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.003655 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.003656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.003771 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.004284 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:57.004292 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:57.005713 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:57.007271 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:57.007276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:57.007278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:57.007481 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:57.007621 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:57.007621 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:57.007621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:57.007734 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:57.007930 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:57.007934 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:57.008047 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 13:17:57.008134 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 13:17:57.008251 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260925 13:17:57.008912 8761 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41827
I20260925 13:17:57.009006 8761 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:57.009189 8761 heartbeater.cc:511] Master 127.0.98.62:41827 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 13:17:57.009742 8570 ts_manager.cc:194] Registered new tserver with Master: a2d69f0e8cc44e43ae4053f2e277d981 (127.0.98.1:34095)
I20260925 13:17:57.010325 8570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:17:57.022460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41827
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:57.101332 8766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:57.101572 8766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:57.101611 8766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:57.102769 8766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:57.102802 8766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:57.102826 8766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:17:57.104350 8766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:57.104557 8766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:57.104727 8766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:57.109313 8772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:57.109308 8774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:57.109308 8771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:57.109661 8766 server_base.cc:1034] running on GCE node
I20260925 13:17:57.109858 8766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:57.110044 8766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:57.111168 8766 hybrid_clock.cc:648] HybridClock initialized: now 1790342277111160 us; error 33 us; skew 500 ppm
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:57.113582 8766 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:17:57.114004 8766 webserver.cc:460] Webserver started at http://127.0.98.2:43827/ using document root <none> and password file <none>
I20260925 13:17:57.114212 8766 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:57.114253 8766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:57.114373 8766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:57.115375 8766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "8a2d436b77a84f859656da61370959f9"
format_stamp: "Formatted at 2026-09-25 13:17:57 on dist-test-slave-t8d9"
server_key: "6fedd7856d97927e13af17226701d625"
server_key_iv: "c1fdee1e5b743c716397ea006005b58e"
server_key_version: "encryptionkey@0"
I20260925 13:17:57.115703 8766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8a2d436b77a84f859656da61370959f9"
format_stamp: "Formatted at 2026-09-25 13:17:57 on dist-test-slave-t8d9"
server_key: "6fedd7856d97927e13af17226701d625"
server_key_iv: "c1fdee1e5b743c716397ea006005b58e"
server_key_version: "encryptionkey@0"
I20260925 13:17:57.116912 8766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:17:57.117501 8781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.117643 8766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.117722 8766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "8a2d436b77a84f859656da61370959f9"
format_stamp: "Formatted at 2026-09-25 13:17:57 on dist-test-slave-t8d9"
server_key: "6fedd7856d97927e13af17226701d625"
server_key_iv: "c1fdee1e5b743c716397ea006005b58e"
server_key_version: "encryptionkey@0"
I20260925 13:17:57.117779 8766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:57.150870 8766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:57.152943 8766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:57.153083 8766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:57.153296 8766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:57.153610 8766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:57.153644 8766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.153672 8766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:57.153692 8766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.159024 8766 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39225
I20260925 13:17:57.159086 8894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39225 every 8 connection(s)
I20260925 13:17:57.159381 8766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:57.167254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 8766
I20260925 13:17:57.167351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:17:57.167518 392 external_mini_cluster.cc:1428] Setting key 45c7fdaf47bdb85439853d084d2bfc0f
I20260925 13:17:57.168082 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:57.160077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33555 (local address 127.0.98.62:41827)
0925 13:17:57.160137 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:57.160139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:57.160327 (+ 188us) server_negotiation.cc:407] Connection header received
0925 13:17:57.161321 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:57.161322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:57.161336 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:57.161389 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:57.162068 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.162568 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.163533 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.163742 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.163798 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.165857 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:57.165858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:57.165859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:57.165866 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:57.166858 (+ 992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:57.167203 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:57.167204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:57.167205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:57.167254 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:57.167476 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:57.167477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:57.167477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:57.167589 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:57.167647 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:57.167912 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 13:17:57.167997 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 13:17:57.168254 8897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:57.160043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:33555)
0925 13:17:57.160208 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:57.160246 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:57.161204 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:57.161481 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:57.161486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:57.161777 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:57.161974 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.161979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.162656 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.162657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.163453 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.163455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.163811 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.163812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.163936 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.164370 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:57.164378 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:57.165776 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:57.166927 (+ 1151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:57.166933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:57.166936 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:57.167141 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:57.167306 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:57.167306 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:57.167307 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:57.167428 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:57.167640 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:57.167646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:57.167793 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 13:17:57.167896 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 13:17:57.168044 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260925 13:17:57.168672 8895 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41827
I20260925 13:17:57.168757 8895 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:57.168918 8895 heartbeater.cc:511] Master 127.0.98.62:41827 requested a full tablet report, sending...
I20260925 13:17:57.169265 8570 ts_manager.cc:194] Registered new tserver with Master: 8a2d436b77a84f859656da61370959f9 (127.0.98.2:39225)
I20260925 13:17:57.169739 8570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33555
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:17:57.189280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41827
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:57.267777 8900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:57.267997 8900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:57.268029 8900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:57.269213 8900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:57.269258 8900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:57.269292 8900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:17:57.270874 8900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:57.271116 8900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:57.271327 8900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:57.275977 8906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:57.275974 8908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:57.276185 8905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:57.276083 8900 server_base.cc:1034] running on GCE node
I20260925 13:17:57.276396 8900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:57.276631 8900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:57.277765 8900 hybrid_clock.cc:648] HybridClock initialized: now 1790342277277750 us; error 28 us; skew 500 ppm
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:57.280078 8900 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:17:57.280499 8900 webserver.cc:460] Webserver started at http://127.0.98.3:35281/ using document root <none> and password file <none>
I20260925 13:17:57.280685 8900 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:57.280725 8900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:57.280820 8900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:17:57.281906 8900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "566f41ad3a7a4520a2674601baf863e5"
format_stamp: "Formatted at 2026-09-25 13:17:57 on dist-test-slave-t8d9"
server_key: "5c61f595da0b7ea9f97638fdd693c2b1"
server_key_iv: "3bf7e3cdc64359f1482e50678eb1ff44"
server_key_version: "encryptionkey@0"
I20260925 13:17:57.282196 8900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "566f41ad3a7a4520a2674601baf863e5"
format_stamp: "Formatted at 2026-09-25 13:17:57 on dist-test-slave-t8d9"
server_key: "5c61f595da0b7ea9f97638fdd693c2b1"
server_key_iv: "3bf7e3cdc64359f1482e50678eb1ff44"
server_key_version: "encryptionkey@0"
I20260925 13:17:57.283350 8900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:17:57.283996 8915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.284185 8900 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:57.284293 8900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "566f41ad3a7a4520a2674601baf863e5"
format_stamp: "Formatted at 2026-09-25 13:17:57 on dist-test-slave-t8d9"
server_key: "5c61f595da0b7ea9f97638fdd693c2b1"
server_key_iv: "3bf7e3cdc64359f1482e50678eb1ff44"
server_key_version: "encryptionkey@0"
I20260925 13:17:57.284379 8900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:57.303658 8900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:57.305464 8900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:57.305624 8900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:57.305872 8900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:57.306198 8900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:57.306243 8900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.306265 8900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:57.306285 8900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.311472 8900 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38345
I20260925 13:17:57.311553 9028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38345 every 8 connection(s)
I20260925 13:17:57.311810 8900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:17:57.313634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 8900
I20260925 13:17:57.313736 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:17:57.313930 392 external_mini_cluster.cc:1428] Setting key 764bdfbff0215483d35c12d7fcb9e89b
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:17:57.320693 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:57.312482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41635 (local address 127.0.98.62:41827)
0925 13:17:57.312545 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:57.312546 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:57.312742 (+ 196us) server_negotiation.cc:407] Connection header received
0925 13:17:57.313782 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:57.313783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:57.313798 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:57.313870 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:57.314654 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.315200 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.316151 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.316348 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.316400 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.318431 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:57.318433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:57.318433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:57.318440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:57.319662 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:57.319962 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:57.319963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:57.319963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:57.320005 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:57.320176 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:57.320177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:57.320177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:57.320294 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:57.320347 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:57.320564 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 13:17:57.320628 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 13:17:57.320887 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:57.312467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:41635)
0925 13:17:57.312628 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:57.312664 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:57.313660 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:57.313943 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:57.313947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:57.314302 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:57.314573 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.314579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.315278 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.315279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.316065 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.316067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.316409 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.316410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.316538 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.316951 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:57.316958 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:57.318338 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:17:57.319720 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:57.319723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:57.319724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:57.319913 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:57.320047 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:17:57.320048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:17:57.320048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:17:57.320137 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:17:57.320349 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:17:57.320354 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:17:57.320470 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 13:17:57.320549 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 13:17:57.320675 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260925 13:17:57.321296 9029 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41827
I20260925 13:17:57.321408 9029 heartbeater.cc:464] Registering TS with master...
I20260925 13:17:57.321604 9029 heartbeater.cc:511] Master 127.0.98.62:41827 requested a full tablet report, sending...
I20260925 13:17:57.321978 8570 ts_manager.cc:194] Registered new tserver with Master: 566f41ad3a7a4520a2674601baf863e5 (127.0.98.3:38345)
I20260925 13:17:57.322326 8570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41635
I20260925 13:17:57.325562 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 13:17:57.387776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 8632
I20260925 13:17:57.391893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 8766
I20260925 13:17:57.395838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 8900
I20260925 13:17:57.401388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 8538
I20260925 13:17:57.406038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41827
--webserver_interface=127.0.98.62
--webserver_port=33487
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:57.484174 9038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:57.484370 9038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:57.484390 9038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:57.485503 9038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:17:57.485563 9038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:57.485590 9038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:57.485615 9038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:17:57.485636 9038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:17:57.487102 9038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:57.487327 9038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:57.487525 9038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:57.491887 9044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:57.491957 9046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:57.491887 9043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:57.492110 9038 server_base.cc:1034] running on GCE node
I20260925 13:17:57.492316 9038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:57.492542 9038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:57.493664 9038 hybrid_clock.cc:648] HybridClock initialized: now 1790342277493653 us; error 25 us; skew 500 ppm
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:57.495853 9038 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:17:57.496275 9038 webserver.cc:460] Webserver started at http://127.0.98.62:33487/ using document root <none> and password file <none>
I20260925 13:17:57.496470 9038 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:57.496526 9038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:57.497799 9038 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:57.498355 9053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.498490 9038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.498564 9038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "3cb2ca7bec12435e87973a1e957d9009"
format_stamp: "Formatted at 2026-09-25 13:17:56 on dist-test-slave-t8d9"
server_key: "d539f3de50c1b25904fb7217fea70441"
server_key_iv: "6d1ed79e76a32166b823b7aa9c515d95"
server_key_version: "encryptionkey@0"
I20260925 13:17:57.498888 9038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:57.516391 9038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:57.518153 9038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:57.518265 9038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:57.521929 9038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41827
I20260925 13:17:57.521960 9105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41827 every 8 connection(s)
I20260925 13:17:57.522469 9038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:17:57.523597 9106 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:17:57.524232 9106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009: Bootstrap starting.
I20260925 13:17:57.530130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 9038
I20260925 13:17:57.530371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34095
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41827
--webserver_port=37827
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:17:57.531883 9106 log.cc:824] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009: Log is configured to *not* fsync() on all Append() calls
I20260925 13:17:57.533576 9106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:17:57.533828 9106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009: Bootstrap complete.
I20260925 13:17:57.535851 9106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb2ca7bec12435e87973a1e957d9009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } }
I20260925 13:17:57.536077 9106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb2ca7bec12435e87973a1e957d9009, State: Initialized, Role: FOLLOWER
I20260925 13:17:57.536237 9106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb2ca7bec12435e87973a1e957d9009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } }
I20260925 13:17:57.536314 9106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:17:57.536362 9106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:17:57.536410 9106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 1 FOLLOWER]: Advancing to term 2
I20260925 13:17:57.537385 9106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb2ca7bec12435e87973a1e957d9009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } }
I20260925 13:17:57.537477 9106 leader_election.cc:302] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cb2ca7bec12435e87973a1e957d9009; no voters:
I20260925 13:17:57.537734 9106 leader_election.cc:288] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 13:17:57.537798 9110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 13:17:57.537986 9110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [term 2 LEADER]: Becoming Leader. State: Replica: 3cb2ca7bec12435e87973a1e957d9009, State: Running, Role: LEADER
I20260925 13:17:57.538079 9106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:17:57.538089 9110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb2ca7bec12435e87973a1e957d9009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } }
I20260925 13:17:57.538498 9112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cb2ca7bec12435e87973a1e957d9009. Latest consensus state: current_term: 2 leader_uuid: "3cb2ca7bec12435e87973a1e957d9009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb2ca7bec12435e87973a1e957d9009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } } }
I20260925 13:17:57.538578 9112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:57.538796 9111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3cb2ca7bec12435e87973a1e957d9009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb2ca7bec12435e87973a1e957d9009" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } } }
I20260925 13:17:57.538893 9117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:17:57.538899 9111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009 [sys.catalog]: This master's current role is: LEADER
I20260925 13:17:57.539430 9117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:17:57.539774 9117 catalog_manager.cc:1259] Loaded cluster ID: c9bf13c48d0f42f69982bf49b24cdc5d
I20260925 13:17:57.539817 9117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:17:57.540617 9117 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:17:57.540789 9117 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3cb2ca7bec12435e87973a1e957d9009: Loaded TSK: 0
I20260925 13:17:57.540995 9117 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 13:17:57.628886 9108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:57.629096 9108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:57.629129 9108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:57.630342 9108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:57.630384 9108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:57.630419 9108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:17:57.631876 9108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37827
--tserver_master_addrs=127.0.98.62:41827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:57.632113 9108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:57.632325 9108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:57.636962 9133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:57.636962 9132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:57.636983 9135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:57.637609 9108 server_base.cc:1034] running on GCE node
I20260925 13:17:57.637837 9108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:57.638052 9108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:57.639180 9108 hybrid_clock.cc:648] HybridClock initialized: now 1790342277639158 us; error 41 us; skew 500 ppm
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342277, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:57.641767 9108 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 13:17:57.642289 9108 webserver.cc:460] Webserver started at http://127.0.98.1:37827/ using document root <none> and password file <none>
I20260925 13:17:57.642549 9108 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:57.642632 9108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:57.644366 9108 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:17:57.644991 9142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.645160 9108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:57.645232 9108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "a2d69f0e8cc44e43ae4053f2e277d981"
format_stamp: "Formatted at 2026-09-25 13:17:56 on dist-test-slave-t8d9"
server_key: "4d3320c138bff8fb5a5e19ee5b57423a"
server_key_iv: "4cff05c239d07c5ffde97659d8d65b92"
server_key_version: "encryptionkey@0"
I20260925 13:17:57.645576 9108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:57.672236 9108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:57.674346 9108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:57.674482 9108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:57.674705 9108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:57.674989 9108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:57.675033 9108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.675062 9108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:57.675079 9108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.680365 9108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34095
I20260925 13:17:57.680415 9255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34095 every 8 connection(s)
I20260925 13:17:57.680728 9108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:17:57.685273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 9108
I20260925 13:17:57.685613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39225
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41827
--webserver_port=43827
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:17:57.689251 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:57.681424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:41789)
0925 13:17:57.681674 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:57.681710 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:57.682153 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:57.682576 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:57.682580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:57.683136 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:57.683323 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.683328 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.684568 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.684568 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.685361 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.685363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.686141 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.686142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.686242 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.686722 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:57.686730 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:57.689058 (+ 2328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":65,"threads_started":1}
W20260925 13:17:57.689268 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:57.681498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41789 (local address 127.0.98.62:41827)
0925 13:17:57.681710 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:57.681719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:57.681784 (+ 65us) server_negotiation.cc:407] Connection header received
0925 13:17:57.682340 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:57.682347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:57.682401 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:57.682551 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:57.683441 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.684488 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.685448 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.686062 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.686165 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.689149 (+ 2984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41789: BlockingRecv error: recv got EOF from 127.0.98.1:41789 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":105,"threads_started":1}
W20260925 13:17:57.689502 9256 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41827 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41827: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:17:57.694079 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:57.689756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:57543)
0925 13:17:57.689797 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:57.689809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:57.689853 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:57.689988 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:57.689989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:57.690133 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:57.690187 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.690189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.690761 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.690762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.691539 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.691542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.691840 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.691841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.691905 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.692272 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:57.692277 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:57.694014 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 13:17:57.694197 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:57.689806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57543 (local address 127.0.98.62:41827)
0925 13:17:57.689841 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:57.689843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:57.689852 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:57.689911 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:57.689912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:57.689921 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:57.689962 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:57.690246 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.690695 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.691610 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.691784 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.691822 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.694101 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57543: BlockingRecv error: recv got EOF from 127.0.98.1:57543 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:17:57.698537 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:57.694388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:48965)
0925 13:17:57.694479 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:57.694490 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:57.694535 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:57.694653 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:57.694654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:57.694781 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:57.694814 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.694816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.695367 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.695367 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.696127 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.696129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.696404 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.696405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.696457 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.696914 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:57.696918 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:57.698471 (+ 1553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":76}
W20260925 13:17:57.698582 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:57.694430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48965 (local address 127.0.98.62:41827)
0925 13:17:57.694459 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:57.694460 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:57.694537 (+ 77us) server_negotiation.cc:407] Connection header received
0925 13:17:57.694579 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:57.694580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:57.694589 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:57.694640 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:57.694874 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.695303 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.696190 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.696351 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.696390 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.698542 (+ 2152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48965: BlockingRecv error: recv got EOF from 127.0.98.1:48965 (error 108)
Metrics: {"server-negotiator.queue_time_us":12}
W20260925 13:17:57.698761 9256 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 13:17:57.766513 9260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:57.766701 9260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:57.766722 9260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:57.767802 9260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:57.767832 9260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:57.767856 9260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:17:57.769186 9260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43827
--tserver_master_addrs=127.0.98.62:41827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:57.769361 9260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:57.769564 9260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:57.773979 9266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:57.773972 9265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:57.773973 9268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:57.774183 9260 server_base.cc:1034] running on GCE node
I20260925 13:17:57.774400 9260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:57.774612 9260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:57.775741 9260 hybrid_clock.cc:648] HybridClock initialized: now 1790342277775726 us; error 32 us; skew 500 ppm
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342277, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:57.778014 9260 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 13:17:57.778378 9260 webserver.cc:460] Webserver started at http://127.0.98.2:43827/ using document root <none> and password file <none>
I20260925 13:17:57.778551 9260 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:57.778596 9260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:57.779752 9260 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:57.780211 9275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.780354 9260 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.780427 9260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "8a2d436b77a84f859656da61370959f9"
format_stamp: "Formatted at 2026-09-25 13:17:57 on dist-test-slave-t8d9"
server_key: "6fedd7856d97927e13af17226701d625"
server_key_iv: "c1fdee1e5b743c716397ea006005b58e"
server_key_version: "encryptionkey@0"
I20260925 13:17:57.780737 9260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:57.806411 9260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:57.808588 9260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:57.808727 9260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:57.808944 9260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:57.809239 9260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:57.809271 9260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.809300 9260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:57.809322 9260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.814641 9260 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39225
I20260925 13:17:57.814721 9388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39225 every 8 connection(s)
I20260925 13:17:57.815011 9260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:17:57.821285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 9260
I20260925 13:17:57.821605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38345
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41827
--webserver_port=35281
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:17:57.821624 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:57.815644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57457 (local address 127.0.98.62:41827)
0925 13:17:57.815687 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:57.815689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:57.815886 (+ 197us) server_negotiation.cc:407] Connection header received
0925 13:17:57.816337 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:57.816338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:57.816348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:57.816406 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:57.817246 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.817773 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.818663 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.818834 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.818874 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.821561 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57457: BlockingRecv error: recv got EOF from 127.0.98.2:57457 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:17:57.821668 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:57.815620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:57457)
0925 13:17:57.815773 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:57.815813 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:57.816267 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:57.816459 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:57.816463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:57.816984 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:57.817181 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.817186 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.817830 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.817830 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.818604 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.818606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.818882 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.818883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.818983 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.819424 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:57.819432 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:57.821445 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
W20260925 13:17:57.821857 9389 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41827 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41827: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:17:57.825831 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:57.822063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:59775)
0925 13:17:57.822108 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:57.822119 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:57.822169 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:57.822277 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:57.822278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:57.822402 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:57.822455 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.822456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.823030 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.823031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.823539 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.823540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.823863 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.823863 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.823926 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.824291 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:57.824296 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:57.825784 (+ 1488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 13:17:57.825951 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:57.822139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59775 (local address 127.0.98.62:41827)
0925 13:17:57.822190 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:57.822192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:57.822201 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:17:57.822222 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:57.822223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:57.822231 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:57.822272 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:57.822511 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.822961 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.823614 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.823808 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.823860 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.825889 (+ 2029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59775: BlockingRecv error: recv got EOF from 127.0.98.2:59775 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:17:57.829694 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:57.826183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:38783)
0925 13:17:57.826223 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:57.826233 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:57.826269 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:57.826369 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:57.826370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:57.826481 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:57.826507 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.826508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.827074 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.827075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.827551 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.827553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.827840 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.827841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.827888 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.828223 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:57.828227 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:57.829629 (+ 1402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 13:17:57.829743 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:57.826241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38783 (local address 127.0.98.62:41827)
0925 13:17:57.826279 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:57.826281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:57.826289 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:17:57.826309 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:57.826310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:57.826318 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:57.826362 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:57.826565 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.827013 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.827620 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.827789 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.827838 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.829692 (+ 1854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38783: BlockingRecv error: recv got EOF from 127.0.98.2:38783 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 13:17:57.829919 9389 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 13:17:57.900906 9392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:17:57.901167 9392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:17:57.901201 9392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:17:57.902407 9392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:17:57.902450 9392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:17:57.902486 9392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:17:57.903919 9392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35281
--tserver_master_addrs=127.0.98.62:41827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:17:57.904175 9392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:17:57.904382 9392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:17:57.908990 9398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:57.909016 9400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:17:57.909021 9397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:17:57.909142 9392 server_base.cc:1034] running on GCE node
I20260925 13:17:57.909384 9392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:17:57.909647 9392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:17:57.910784 9392 hybrid_clock.cc:648] HybridClock initialized: now 1790342277910769 us; error 29 us; skew 500 ppm
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342277, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:17:57.913596 9392 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 13:17:57.914101 9392 webserver.cc:460] Webserver started at http://127.0.98.3:35281/ using document root <none> and password file <none>
I20260925 13:17:57.914324 9392 fs_manager.cc:359] Metadata directory not provided
I20260925 13:17:57.914382 9392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:17:57.915768 9392 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:17:57.916296 9407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.916441 9392 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:17:57.916517 9392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "566f41ad3a7a4520a2674601baf863e5"
format_stamp: "Formatted at 2026-09-25 13:17:57 on dist-test-slave-t8d9"
server_key: "5c61f595da0b7ea9f97638fdd693c2b1"
server_key_iv: "3bf7e3cdc64359f1482e50678eb1ff44"
server_key_version: "encryptionkey@0"
I20260925 13:17:57.916850 9392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:17:57.946753 9392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:17:57.948648 9392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:17:57.948783 9392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:17:57.949003 9392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:17:57.949302 9392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:17:57.949333 9392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.949362 9392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:17:57.949379 9392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:17:57.954874 9392 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38345
I20260925 13:17:57.954936 9520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38345 every 8 connection(s)
I20260925 13:17:57.955245 9392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:17:57.956422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 9392
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 13:17:57.962535 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:57.955864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37587 (local address 127.0.98.62:41827)
0925 13:17:57.955931 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:57.955933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:57.956135 (+ 202us) server_negotiation.cc:407] Connection header received
0925 13:17:57.956651 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:57.956652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:57.956665 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:57.956711 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:57.957886 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.958351 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.959299 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.959566 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.959639 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.962465 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37587: BlockingRecv error: recv got EOF from 127.0.98.3:37587 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 13:17:57.962599 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:57.955854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:37587)
0925 13:17:57.955995 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:57.956036 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:57.956518 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:57.956791 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:57.956795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:57.957506 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:57.957817 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.957824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.958430 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.958431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.959225 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.959227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.959651 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.959652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.959777 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.960272 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:57.960281 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:57.962392 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
W20260925 13:17:57.962818 9521 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41827 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41827: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260925 13:17:57.964586 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:57.956907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.0.98.62:41827)
0925 13:17:57.957042 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:57.957043 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:57.957053 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:17:57.957076 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:57.957077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:57.957085 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:57.957140 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:57.957836 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.958348 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.960043 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.960336 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.960396 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.961666 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:17:57.961670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:17:57.961670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:17:57.961680 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:17:57.963372 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:57.963711 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:57.963713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:57.963713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:57.963803 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:17:57.963987 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:17:57.963988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:17:57.963988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:17:57.964188 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:17:57.964314 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 13:17:57.964386 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 13:17:57.964491 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:17:57.967125 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:57.963020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:36199)
0925 13:17:57.963120 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:57.963130 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:57.963182 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:57.963296 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:57.963296 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:57.963426 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:57.963483 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.963484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.964217 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.964218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.964875 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.964878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.965260 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.965261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.965322 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.965676 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:57.965680 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:57.967062 (+ 1382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260925 13:17:57.967204 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:57.963076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36199 (local address 127.0.98.62:41827)
0925 13:17:57.963110 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:57.963112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:57.963167 (+ 55us) server_negotiation.cc:407] Connection header received
0925 13:17:57.963234 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:57.963234 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:57.963241 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:57.963289 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:57.963549 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.964151 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.964939 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.965192 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.965252 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.967154 (+ 1902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36199: BlockingRecv error: recv got EOF from 127.0.98.3:36199 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:57 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:17:57.970672 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:57.967389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:57033)
0925 13:17:57.967431 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:57.967440 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:57.967473 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:57.967565 (+ 92us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:57.967566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:57.967669 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:57.967694 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.967695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.968217 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.968218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.968675 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:57.968676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.968922 (+ 246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.968923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:57.968954 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.969248 (+ 294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:57.969251 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:57.970598 (+ 1347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 13:17:57.970742 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:57.967435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57033 (local address 127.0.98.62:41827)
0925 13:17:57.967474 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:57.967476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:57.967504 (+ 28us) server_negotiation.cc:407] Connection header received
0925 13:17:57.967521 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:57.967522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:57.967528 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:57.967562 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:57.967739 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.968170 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.968723 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:57.968882 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:57.968919 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:57.970673 (+ 1754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57033: BlockingRecv error: recv got EOF from 127.0.98.3:57033 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 13:17:57.971004 9521 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 13:17:58 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:58 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:17:58.704732 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:58.699112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:52797)
0925 13:17:58.699241 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:58.699251 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:58.699302 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:58.699454 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:58.699455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:58.699660 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:58.699716 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:58.699719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:58.700644 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:58.700645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:58.701469 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:58.701473 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:58.701974 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:58.701975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:58.702036 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:58.702546 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:58.702552 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:58.704636 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
W20260925 13:17:58.704977 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:58.699173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52797 (local address 127.0.98.62:41827)
0925 13:17:58.699295 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:58.699297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:58.699307 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:17:58.699354 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:58.699355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:58.699367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:58.699447 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:58.699785 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:58.700549 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:58.701578 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:58.701884 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:58.701962 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:58.704915 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52797: BlockingRecv error: recv got EOF from 127.0.98.1:52797 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
Sep 25 13:17:58 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:58 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:17:58.834364 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:58.830334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:36889)
0925 13:17:58.830481 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:58.830493 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:58.830557 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:58.830678 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:58.830678 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:58.830828 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:58.830874 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:58.830875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:58.831449 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:58.831450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:58.831995 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:58.831997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:58.832280 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:58.832281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:58.832316 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:58.832728 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:58.832732 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:58.834296 (+ 1564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
W20260925 13:17:58.834447 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:58.830429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36889 (local address 127.0.98.62:41827)
0925 13:17:58.830490 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:58.830491 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:58.830541 (+ 50us) server_negotiation.cc:407] Connection header received
0925 13:17:58.830612 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:58.830613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:58.830624 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:58.830662 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:58.830922 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:58.831398 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:58.832058 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:58.832232 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:58.832273 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:58.834390 (+ 2117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36889: BlockingRecv error: recv got EOF from 127.0.98.2:36889 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 13:17:58 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:58 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:17:58.975498 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:58.971387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:47367)
0925 13:17:58.971508 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:58.971519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:58.971581 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:58.971732 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:58.971733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:58.971903 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:58.971951 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:58.971953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:58.972548 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:58.972548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:58.973091 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:58.973093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:58.973384 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:58.973385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:58.973427 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:58.973841 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:58.973845 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:58.975410 (+ 1565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
W20260925 13:17:58.975569 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:58.971468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47367 (local address 127.0.98.62:41827)
0925 13:17:58.971521 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:58.971523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:58.971580 (+ 57us) server_negotiation.cc:407] Connection header received
0925 13:17:58.971647 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:58.971648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:58.971660 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:58.971699 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:58.972009 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:58.972491 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:58.973165 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:58.973336 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:58.973378 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:58.975515 (+ 2137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47367: BlockingRecv error: recv got EOF from 127.0.98.3:47367 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 13:17:59 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:59 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:17:59.714445 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:59.709965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:33163)
0925 13:17:59.710093 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:59.710104 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:59.710153 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:59.710323 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:59.710324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:59.710474 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:59.710521 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:59.710523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:59.711173 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:59.711174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:59.711697 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:59.711698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:59.712148 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:59.712149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:59.712207 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:59.712741 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:59.712745 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:59.714388 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
W20260925 13:17:59.714577 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:59.710068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33163 (local address 127.0.98.62:41827)
0925 13:17:59.710198 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:59.710200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:59.710208 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:17:59.710235 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:59.710235 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:59.710244 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:59.710313 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:59.710595 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:59.711096 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:59.711809 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:59.712070 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:59.712149 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:59.714497 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33163: BlockingRecv error: recv got EOF from 127.0.98.1:33163 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
Sep 25 13:17:59 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:59 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:17:59.838992 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:59.834864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:44681)
0925 13:17:59.835006 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:59.835016 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:59.835078 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:59.835185 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:59.835185 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:59.835339 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:59.835384 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:59.835386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:59.835980 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:59.835981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:59.836498 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:59.836500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:59.836784 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:59.836785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:59.836821 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:59.837214 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:59.837217 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:59.838934 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
W20260925 13:17:59.839109 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:59.834948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44681 (local address 127.0.98.62:41827)
0925 13:17:59.835007 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:59.835009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:59.835063 (+ 54us) server_negotiation.cc:407] Connection header received
0925 13:17:59.835119 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:59.835120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:59.835130 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:59.835173 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:59.835434 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:59.835922 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:59.836555 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:59.836736 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:59.836778 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:59.839022 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44681: BlockingRecv error: recv got EOF from 127.0.98.2:44681 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 25 13:17:59 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:17:59 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:17:59.980461 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:17:59.975954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:53619)
0925 13:17:59.976076 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 13:17:59.976087 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:17:59.976133 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:17:59.976333 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:17:59.976334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:17:59.976495 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:17:59.976543 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:59.976545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:59.977186 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:59.977187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:59.977735 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:17:59.977737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:17:59.978051 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:17:59.978052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:17:59.978104 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:59.978570 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:17:59.978574 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:17:59.980384 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":54,"threads_started":1}
W20260925 13:17:59.980537 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:17:59.976158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53619 (local address 127.0.98.62:41827)
0925 13:17:59.976225 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 13:17:59.976226 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:17:59.976233 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:17:59.976252 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:17:59.976253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:17:59.976265 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:17:59.976309 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:17:59.976616 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:59.977104 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:59.977808 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:17:59.977991 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:17:59.978046 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:17:59.980484 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53619: BlockingRecv error: recv got EOF from 127.0.98.3:53619 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 13:18:00 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:00 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:00.726940 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:00.714873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:52983)
0925 13:18:00.715017 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:00.715030 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:00.715090 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:00.715209 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:00.715209 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:00.715365 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:00.715414 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:00.715416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:00.716061 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:00.716062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:00.716594 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:00.716595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:00.716901 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:00.716901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:00.716936 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:00.724891 (+ 7955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:00.724900 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:00.726870 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
W20260925 13:18:00.727129 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:00.714972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52983 (local address 127.0.98.62:41827)
0925 13:18:00.715097 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:00.715099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:00.715108 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:18:00.715130 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:00.715131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:00.715142 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:00.715194 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:00.715472 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:00.715990 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:00.716668 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:00.716852 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:00.716897 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:00.727067 (+ 10170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52983: BlockingRecv error: recv got EOF from 127.0.98.1:52983 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
Sep 25 13:18:00 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:00 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:00.843621 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:00.839462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:44191)
0925 13:18:00.839592 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:00.839603 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:00.839681 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:00.839795 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:00.839796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:00.839962 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:00.840012 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:00.840014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:00.840598 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:00.840598 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:00.841116 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:00.841118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:00.841433 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:00.841434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:00.841470 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:00.841924 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:00.841927 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:00.843552 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
W20260925 13:18:00.843758 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:00.839526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44191 (local address 127.0.98.62:41827)
0925 13:18:00.839589 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:00.839592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:00.839672 (+ 80us) server_negotiation.cc:407] Connection header received
0925 13:18:00.839739 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:00.839740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:00.839751 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:00.839788 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:00.840061 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:00.840543 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:00.841176 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:00.841375 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:00.841427 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:00.843648 (+ 2221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44191: BlockingRecv error: recv got EOF from 127.0.98.2:44191 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 13:18:00 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:00 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:00.985502 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:00.981038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:34599)
0925 13:18:00.981171 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:00.981184 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:00.981256 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:00.981391 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:00.981392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:00.981621 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:00.981670 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:00.981672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:00.982353 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:00.982354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:00.982902 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:00.982904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:00.983263 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:00.983264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:00.983319 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:00.983779 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:00.983784 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:00.985435 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
W20260925 13:18:00.985685 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:00.981095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34599 (local address 127.0.98.62:41827)
0925 13:18:00.981175 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:00.981178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:00.981246 (+ 68us) server_negotiation.cc:407] Connection header received
0925 13:18:00.981318 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:00.981319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:00.981334 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:00.981390 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:00.981739 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:00.982271 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:00.982976 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:00.983204 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:00.983258 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:00.985559 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34599: BlockingRecv error: recv got EOF from 127.0.98.3:34599 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 13:18:01 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:01 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:01.731794 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:01.727443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:38069)
0925 13:18:01.727585 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:01.727597 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:01.727658 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:01.727806 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:01.727806 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:01.727975 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:01.728020 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:01.728022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:01.728659 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:01.728660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:01.729168 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:01.729177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:01.729495 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:01.729496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:01.729579 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:01.730029 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:01.730033 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:01.731734 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
W20260925 13:18:01.731938 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:01.727486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38069 (local address 127.0.98.62:41827)
0925 13:18:01.727596 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:01.727598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:01.727656 (+ 58us) server_negotiation.cc:407] Connection header received
0925 13:18:01.727723 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:01.727724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:01.727736 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:01.727791 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:01.728092 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:01.728584 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:01.729247 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:01.729437 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:01.729490 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:01.731866 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38069: BlockingRecv error: recv got EOF from 127.0.98.1:38069 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
Sep 25 13:18:01 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:01 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:01.848817 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:01.844195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:56705)
0925 13:18:01.844341 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:01.844352 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:01.844415 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:01.844565 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:01.844565 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:01.844732 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:01.844780 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:01.844782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:01.845435 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:01.845437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:01.846031 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:01.846033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:01.846504 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:01.846505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:01.846558 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:01.846987 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:01.846991 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:01.848739 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":59,"threads_started":1}
W20260925 13:18:01.848908 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:01.844289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56705 (local address 127.0.98.62:41827)
0925 13:18:01.844353 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:01.844355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:01.844438 (+ 83us) server_negotiation.cc:407] Connection header received
0925 13:18:01.844475 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:01.844476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:01.844489 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:01.844553 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:01.844835 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:01.845343 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:01.846168 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:01.846394 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:01.846456 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:01.848855 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56705: BlockingRecv error: recv got EOF from 127.0.98.2:56705 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 13:18:01 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:01 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:01.990403 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:01.985989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:47141)
0925 13:18:01.986213 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:01.986227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:01.986306 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:01.986454 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:01.986455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:01.986625 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:01.986673 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:01.986675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:01.987342 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:01.987343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:01.987868 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:01.987870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:01.988193 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:01.988195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:01.988249 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:01.988674 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:01.988678 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:01.990342 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":112,"threads_started":1}
W20260925 13:18:01.990496 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:01.986100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47141 (local address 127.0.98.62:41827)
0925 13:18:01.986217 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:01.986220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:01.986298 (+ 78us) server_negotiation.cc:407] Connection header received
0925 13:18:01.986370 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:01.986372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:01.986390 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:01.986437 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:01.986752 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:01.987268 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:01.987944 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:01.988134 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:01.988187 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:01.990434 (+ 2247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47141: BlockingRecv error: recv got EOF from 127.0.98.3:47141 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 13:18:02 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:02 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:02.736753 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:02.732295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:41723)
0925 13:18:02.732432 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:02.732443 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:02.732491 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:02.732617 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:02.732618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:02.732772 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:02.732820 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:02.732821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:02.733456 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:02.733457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:02.734074 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:02.734077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:02.734391 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:02.734392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:02.734433 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:02.734884 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:02.734889 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:02.736691 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
W20260925 13:18:02.736845 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:02.732378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41723 (local address 127.0.98.62:41827)
0925 13:18:02.732484 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:02.732486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:02.732524 (+ 38us) server_negotiation.cc:407] Connection header received
0925 13:18:02.732546 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:02.732547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:02.732556 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:02.732602 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:02.732874 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:02.733371 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:02.734144 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:02.734334 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:02.734382 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:02.736780 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41723: BlockingRecv error: recv got EOF from 127.0.98.1:41723 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
Sep 25 13:18:02 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:02 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:02.853471 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:02.849286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:38263)
0925 13:18:02.849410 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:02.849420 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:02.849479 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:02.849629 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:02.849630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:02.849779 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:02.849827 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:02.849829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:02.850439 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:02.850440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:02.850988 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:02.850990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:02.851365 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:02.851366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:02.851401 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:02.851834 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:02.851839 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:02.853405 (+ 1566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
W20260925 13:18:02.853577 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:02.849378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38263 (local address 127.0.98.62:41827)
0925 13:18:02.849437 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:02.849439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:02.849475 (+ 36us) server_negotiation.cc:407] Connection header received
0925 13:18:02.849520 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:02.849520 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:02.849554 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:02.849615 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:02.849874 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:02.850371 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:02.851106 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:02.851300 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:02.851359 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:02.853494 (+ 2135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38263: BlockingRecv error: recv got EOF from 127.0.98.2:38263 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 13:18:02 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:02 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:02.995064 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:02.990864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:56541)
0925 13:18:02.991010 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:02.991023 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:02.991097 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:02.991217 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:02.991218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:02.991385 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:02.991429 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:02.991431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:02.992055 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:02.992057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:02.992597 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:02.992599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:02.992908 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:02.992909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:02.992962 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:02.993358 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:02.993362 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:02.995004 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":59,"threads_started":1}
W20260925 13:18:02.995217 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:02.990963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56541 (local address 127.0.98.62:41827)
0925 13:18:02.991052 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:02.991054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:02.991076 (+ 22us) server_negotiation.cc:407] Connection header received
0925 13:18:02.991148 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:02.991148 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:02.991161 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:02.991206 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:02.991501 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:02.991979 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:02.992671 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:02.992851 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:02.992905 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:02.995151 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56541: BlockingRecv error: recv got EOF from 127.0.98.3:56541 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 13:18:03 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:03 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:03.741710 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:03.737288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:57387)
0925 13:18:03.737417 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:03.737427 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:03.737479 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:03.737679 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:03.737680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:03.737831 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:03.737878 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:03.737880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:03.738524 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:03.738526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:03.739101 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:03.739103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:03.739437 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:03.739437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:03.739472 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:03.739900 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:03.739905 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:03.741620 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
W20260925 13:18:03.741777 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:03.737374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57387 (local address 127.0.98.62:41827)
0925 13:18:03.737523 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:03.737525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:03.737553 (+ 28us) server_negotiation.cc:407] Connection header received
0925 13:18:03.737580 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:03.737581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:03.737593 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:03.737667 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:03.737935 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:03.738432 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:03.739173 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:03.739387 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:03.739433 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:03.741718 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57387: BlockingRecv error: recv got EOF from 127.0.98.1:57387 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":56,"threads_started":1}
Sep 25 13:18:03 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:03 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:03.858129 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:03.853965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:57763)
0925 13:18:03.854110 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:03.854154 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:03.854220 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:03.854335 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:03.854335 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:03.854488 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:03.854534 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:03.854535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:03.855140 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:03.855141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:03.855652 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:03.855653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:03.855929 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:03.855930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:03.855965 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:03.856407 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:03.856411 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:03.858067 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
W20260925 13:18:03.858227 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:03.854043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57763 (local address 127.0.98.62:41827)
0925 13:18:03.854102 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:03.854104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:03.854212 (+ 108us) server_negotiation.cc:407] Connection header received
0925 13:18:03.854265 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:03.854265 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:03.854276 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:03.854321 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:03.854581 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:03.855065 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:03.855708 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:03.855882 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:03.855925 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:03.858169 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57763: BlockingRecv error: recv got EOF from 127.0.98.2:57763 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 13:18:03 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:03 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:03.999820 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:03.995509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:60467)
0925 13:18:03.995656 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:03.995670 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:03.995745 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:03.995875 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:03.995876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:03.996038 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:03.996103 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:03.996106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:03.996775 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:03.996776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:03.997301 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:03.997303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:03.997648 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:03.997649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:03.997717 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:03.998150 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:03.998154 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:03.999733 (+ 1579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
W20260925 13:18:03.999882 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:03.995596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60467 (local address 127.0.98.62:41827)
0925 13:18:03.995677 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:03.995679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:03.995736 (+ 57us) server_negotiation.cc:407] Connection header received
0925 13:18:03.995804 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:03.995805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:03.995819 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:03.995865 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:03.996177 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:03.996700 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:03.997374 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:03.997595 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:03.997649 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:03.999829 (+ 2180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60467: BlockingRecv error: recv got EOF from 127.0.98.3:60467 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 13:18:04 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:04 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:04.746713 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:04.742286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:37841)
0925 13:18:04.742402 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:04.742410 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:04.742454 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:04.742691 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:04.742692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:04.742857 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:04.742908 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:04.742910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:04.743564 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:04.743565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:04.744149 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:04.744151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:04.744517 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:04.744517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:04.744557 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:04.744973 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:04.744978 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:04.746652 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
W20260925 13:18:04.746840 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:04.742398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37841 (local address 127.0.98.62:41827)
0925 13:18:04.742528 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:04.742530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:04.742539 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:18:04.742565 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:04.742566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:04.742576 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:04.742641 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:04.742972 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:04.743485 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:04.744256 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:04.744456 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:04.744513 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:04.746762 (+ 2249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37841: BlockingRecv error: recv got EOF from 127.0.98.1:37841 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
Sep 25 13:18:04 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:04 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:04.863083 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:04.858522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:60959)
0925 13:18:04.858652 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:04.858663 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:04.858729 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:04.858855 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:04.858856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:04.859029 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:04.859075 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:04.859077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:04.859727 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:04.859728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:04.860257 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:04.860259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:04.860592 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:04.860593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:04.860656 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:04.861118 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:04.861124 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:04.863000 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
W20260925 13:18:04.863219 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:04.858566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60959 (local address 127.0.98.62:41827)
0925 13:18:04.858614 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:04.858616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:04.858727 (+ 111us) server_negotiation.cc:407] Connection header received
0925 13:18:04.858782 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:04.858782 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:04.858796 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:04.858842 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:04.859154 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:04.859649 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:04.860345 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:04.860533 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:04.860588 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:04.863164 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60959: BlockingRecv error: recv got EOF from 127.0.98.2:60959 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 13:18:05 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:05 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:05.004758 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:05.000275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:48923)
0925 13:18:05.000410 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:05.000422 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:05.000489 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:05.000613 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:05.000613 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:05.000780 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:05.000825 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:05.000827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:05.001482 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:05.001484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:05.002060 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:05.002062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:05.002408 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:05.002409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:05.002464 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:05.002896 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:05.002900 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:05.004686 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
W20260925 13:18:05.004844 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:05.000347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48923 (local address 127.0.98.62:41827)
0925 13:18:05.000403 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:05.000405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:05.000475 (+ 70us) server_negotiation.cc:407] Connection header received
0925 13:18:05.000537 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:05.000537 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:05.000552 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:05.000600 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:05.000893 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:05.001412 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:05.002167 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:05.002349 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:05.002403 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:05.004785 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48923: BlockingRecv error: recv got EOF from 127.0.98.3:48923 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 13:18:05 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:05 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:05.756422 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:05.747228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:49225)
0925 13:18:05.747390 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:05.747400 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:05.747453 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:05.747576 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:05.747577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:05.747738 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:05.747784 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:05.747786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:05.748484 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:05.748485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:05.749034 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:05.749036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:05.749348 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:05.749349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:05.749391 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:05.754616 (+ 5225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:05.754621 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:05.756362 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":76,"threads_started":1}
W20260925 13:18:05.756551 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:05.747347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49225 (local address 127.0.98.62:41827)
0925 13:18:05.747443 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:05.747444 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:05.747454 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:18:05.747499 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:05.747500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:05.747509 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:05.747566 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:05.747854 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:05.748415 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:05.749108 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:05.749296 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:05.749340 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:05.756472 (+ 7132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49225: BlockingRecv error: recv got EOF from 127.0.98.1:49225 (error 108)
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":53,"threads_started":1}
Sep 25 13:18:05 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:05 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:05.868425 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:05.863552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:45207)
0925 13:18:05.863708 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:05.863722 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:05.863826 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:05.863996 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:05.863997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:05.864193 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:05.864243 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:05.864245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:05.865095 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:05.865097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:05.865695 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:05.865698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:05.866062 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:05.866063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:05.866118 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:05.866594 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:05.866599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:05.868346 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
W20260925 13:18:05.868513 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:05.863623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45207 (local address 127.0.98.62:41827)
0925 13:18:05.863716 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:05.863718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:05.863805 (+ 87us) server_negotiation.cc:407] Connection header received
0925 13:18:05.863909 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:05.863910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:05.863925 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:05.863985 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:05.864325 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:05.865000 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:05.865780 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:05.865991 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:05.866057 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:05.868459 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45207: BlockingRecv error: recv got EOF from 127.0.98.2:45207 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 13:18:06 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:06 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:06.009518 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:06.005233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:39591)
0925 13:18:06.005404 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:06.005417 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:06.005487 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:06.005638 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:06.005639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:06.005792 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:06.005838 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:06.005840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:06.006463 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:06.006464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:06.006999 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:06.007000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:06.007310 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:06.007311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:06.007347 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:06.007752 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:06.007756 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:06.009448 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
W20260925 13:18:06.009697 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:06.005348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39591 (local address 127.0.98.62:41827)
0925 13:18:06.005411 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:06.005414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:06.005470 (+ 56us) server_negotiation.cc:407] Connection header received
0925 13:18:06.005552 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:06.005553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:06.005568 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:06.005626 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:06.005892 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:06.006404 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:06.007061 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:06.007258 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:06.007305 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:06.009628 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39591: BlockingRecv error: recv got EOF from 127.0.98.3:39591 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 13:18:06 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:06 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:06.761258 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:06.756949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:37377)
0925 13:18:06.757117 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:06.757130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:06.757216 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:06.757350 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:06.757351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:06.757506 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:06.757600 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:06.757602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:06.758232 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:06.758232 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:06.758744 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:06.758745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:06.759096 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:06.759097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:06.759146 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:06.759550 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:06.759553 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:06.761201 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
W20260925 13:18:06.761358 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:06.757048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37377 (local address 127.0.98.62:41827)
0925 13:18:06.757171 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:06.757173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:06.757200 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:18:06.757266 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:06.757267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:06.757278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:06.757333 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:06.757666 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:06.758172 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:06.758823 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:06.759024 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:06.759076 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:06.761287 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37377: BlockingRecv error: recv got EOF from 127.0.98.1:37377 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
Sep 25 13:18:06 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:06 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:06.873646 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:06.869007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:34589)
0925 13:18:06.869222 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:06.869234 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:06.869307 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:06.869434 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:06.869435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:06.869648 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:06.869716 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:06.869719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:06.870578 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:06.870579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:06.871101 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:06.871103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:06.871448 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:06.871448 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:06.871497 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:06.871930 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:06.871933 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:06.873584 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":121,"threads_started":1}
W20260925 13:18:06.873734 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:06.869108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34589 (local address 127.0.98.62:41827)
0925 13:18:06.869176 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:06.869178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:06.869298 (+ 120us) server_negotiation.cc:407] Connection header received
0925 13:18:06.869355 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:06.869356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:06.869368 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:06.869416 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:06.869793 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:06.870514 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:06.871197 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:06.871387 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:06.871443 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:06.873653 (+ 2210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34589: BlockingRecv error: recv got EOF from 127.0.98.2:34589 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 13:18:07 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:07 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:07.014530 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:07.010078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:45611)
0925 13:18:07.010237 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:07.010247 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:07.010289 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:07.010414 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:07.010415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:07.010568 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:07.010614 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:07.010616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:07.011303 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:07.011304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:07.011936 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:07.011939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:07.012242 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:07.012243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:07.012279 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:07.012689 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:07.012693 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:07.014461 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
W20260925 13:18:07.014639 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:07.010200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45611 (local address 127.0.98.62:41827)
0925 13:18:07.010263 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:07.010265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:07.010304 (+ 39us) server_negotiation.cc:407] Connection header received
0925 13:18:07.010333 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:07.010335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:07.010346 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:07.010388 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:07.010671 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:07.011213 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:07.012001 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:07.012185 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:07.012237 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:07.014572 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45611: BlockingRecv error: recv got EOF from 127.0.98.3:45611 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 13:18:07 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:07 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:07.765794 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:07.761703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:39457)
0925 13:18:07.761831 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:07.761842 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:07.761897 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:07.762029 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:07.762030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:07.762186 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:07.762232 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:07.762234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:07.762865 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:07.762865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:07.763376 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:07.763377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:07.763658 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:07.763658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:07.763692 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:07.764122 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:07.764126 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:07.765727 (+ 1601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
W20260925 13:18:07.765898 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:07.761774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39457 (local address 127.0.98.62:41827)
0925 13:18:07.761910 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:07.761912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:07.761923 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:18:07.761946 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:07.761947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:07.761957 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:07.762020 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:07.762301 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:07.762809 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:07.763433 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:07.763611 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:07.763652 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:07.765830 (+ 2178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39457: BlockingRecv error: recv got EOF from 127.0.98.1:39457 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":54,"threads_started":1}
Sep 25 13:18:07 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:07 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:07.878417 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:07.874122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:36007)
0925 13:18:07.874268 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:07.874280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:07.874357 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:07.874488 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:07.874489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:07.874659 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:07.874705 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:07.874706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:07.875336 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:07.875337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:07.875864 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:07.875865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:07.876180 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:07.876181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:07.876234 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:07.876669 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:07.876673 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:07.878342 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
W20260925 13:18:07.878486 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:07.874197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36007 (local address 127.0.98.62:41827)
0925 13:18:07.874269 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:07.874271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:07.874344 (+ 73us) server_negotiation.cc:407] Connection header received
0925 13:18:07.874416 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:07.874417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:07.874429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:07.874477 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:07.874776 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:07.875265 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:07.875935 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:07.876123 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:07.876177 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:07.878435 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36007: BlockingRecv error: recv got EOF from 127.0.98.2:36007 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 13:18:08 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:08 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:08.019731 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:08.015024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:39763)
0925 13:18:08.015158 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:08.015172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:08.015241 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:08.015366 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:08.015367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:08.015519 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:08.015566 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:08.015567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:08.016149 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:08.016150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:08.016693 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:08.016695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:08.017116 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:08.017117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:08.017178 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:08.017685 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:08.017691 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:08.019671 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
W20260925 13:18:08.019819 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:08.015181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39763 (local address 127.0.98.62:41827)
0925 13:18:08.015265 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:08.015266 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:08.015274 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:18:08.015296 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:08.015296 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:08.015309 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:08.015355 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:08.015618 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:08.016094 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:08.016788 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:08.017040 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:08.017106 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:08.019768 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39763: BlockingRecv error: recv got EOF from 127.0.98.3:39763 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 13:18:08 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:08 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:08.771176 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:08.766201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:50679)
0925 13:18:08.766327 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:08.766338 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:08.766383 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:08.766529 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:08.766530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:08.766686 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:08.766732 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:08.766734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:08.767659 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:08.767660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:08.768170 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:08.768171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:08.768626 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:08.768627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:08.768663 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:08.769154 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:08.769158 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:08.771093 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
W20260925 13:18:08.771272 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:08.766259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50679 (local address 127.0.98.62:41827)
0925 13:18:08.766386 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:08.766388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:08.766398 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:18:08.766427 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:08.766428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:08.766441 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:08.766527 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:08.766798 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:08.767585 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:08.768247 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:08.768565 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:08.768630 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:08.771211 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50679: BlockingRecv error: recv got EOF from 127.0.98.1:50679 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
Sep 25 13:18:08 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:08 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:08.883301 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:08.878974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:52877)
0925 13:18:08.879115 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:08.879126 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:08.879186 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:08.879343 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:08.879343 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:08.879493 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:08.879540 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:08.879542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:08.880133 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:08.880134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:08.880656 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:08.880658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:08.880932 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:08.880932 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:08.880966 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:08.881388 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:08.881392 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:08.883237 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
W20260925 13:18:08.883392 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:08.879064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52877 (local address 127.0.98.62:41827)
0925 13:18:08.879163 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:08.879164 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:08.879175 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:18:08.879268 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:08.879269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:08.879281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:08.879327 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:08.879590 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:08.880074 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:08.880712 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:08.880885 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:08.880927 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:08.883333 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52877: BlockingRecv error: recv got EOF from 127.0.98.2:52877 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 13:18:09 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:09 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:09.024808 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:09.020239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:49207)
0925 13:18:09.020373 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:09.020385 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:09.020461 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:09.020586 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:09.020587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:09.020774 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:09.020819 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:09.020821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:09.021567 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:09.021568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:09.022095 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:09.022097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:09.022458 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:09.022459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:09.022512 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:09.022901 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:09.022905 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:09.024748 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
W20260925 13:18:09.024892 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:09.020319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49207 (local address 127.0.98.62:41827)
0925 13:18:09.020376 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:09.020378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:09.020441 (+ 63us) server_negotiation.cc:407] Connection header received
0925 13:18:09.020514 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:09.020515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:09.020528 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:09.020572 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:09.020895 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:09.021463 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:09.022190 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:09.022398 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:09.022452 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:09.024831 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49207: BlockingRecv error: recv got EOF from 127.0.98.3:49207 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 13:18:09 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:09 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:09.775900 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:09.771618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:45883)
0925 13:18:09.771745 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:09.771756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:09.771806 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:09.771922 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:09.771923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:09.772073 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:09.772121 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:09.772123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:09.772780 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:09.772781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:09.773334 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:09.773336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:09.773733 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:09.773734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:09.773784 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:09.774199 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:09.774203 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:09.775838 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
W20260925 13:18:09.776002 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:09.771689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45883 (local address 127.0.98.62:41827)
0925 13:18:09.771809 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:09.771811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:09.771819 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:18:09.771856 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:09.771856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:09.771866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:09.771912 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:09.772173 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:09.772688 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:09.773433 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:09.773663 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:09.773717 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:09.775934 (+ 2217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45883: BlockingRecv error: recv got EOF from 127.0.98.1:45883 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
Sep 25 13:18:09 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:09 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:09.888329 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:09.883807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:56645)
0925 13:18:09.883950 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:09.883959 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:09.884018 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:09.884196 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:09.884197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:09.884385 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:09.884440 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:09.884442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:09.885161 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:09.885162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:09.885741 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:09.885744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:09.886103 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:09.886104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:09.886152 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:09.886578 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:09.886582 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:09.888255 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
W20260925 13:18:09.888394 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:09.883891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56645 (local address 127.0.98.62:41827)
0925 13:18:09.883945 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:09.883947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:09.884055 (+ 108us) server_negotiation.cc:407] Connection header received
0925 13:18:09.884083 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:09.884083 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:09.884094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:09.884136 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:09.884528 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:09.885073 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:09.885802 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:09.885994 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:09.886039 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:09.888345 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56645: BlockingRecv error: recv got EOF from 127.0.98.2:56645 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 13:18:10 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:10 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:10.029913 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:10.025347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:58055)
0925 13:18:10.025472 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:10.025482 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:10.025573 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:10.025715 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:10.025716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:10.025883 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:10.025933 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:10.025935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:10.026623 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:10.026623 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:10.027172 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:10.027173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:10.027499 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:10.027500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:10.027555 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:10.027989 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:10.027993 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:10.029844 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
W20260925 13:18:10.029996 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:10.025452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58055 (local address 127.0.98.62:41827)
0925 13:18:10.025521 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:10.025523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:10.025552 (+ 29us) server_negotiation.cc:407] Connection header received
0925 13:18:10.025635 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:10.025636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:10.025648 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:10.025703 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:10.026032 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:10.026546 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:10.027243 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:10.027422 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:10.027471 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:10.029934 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58055: BlockingRecv error: recv got EOF from 127.0.98.3:58055 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 13:18:10 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:10 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:10.786471 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:10.776491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:50539)
0925 13:18:10.776655 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:10.776677 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:10.776737 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:10.776864 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:10.776865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:10.777019 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:10.777065 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:10.777066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:10.777768 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:10.777769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:10.778293 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:10.778295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:10.778593 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:10.778594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:10.778630 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:10.784455 (+ 5825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:10.784475 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:10.786373 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
W20260925 13:18:10.786561 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:10.776613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50539 (local address 127.0.98.62:41827)
0925 13:18:10.776711 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:10.776713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:10.776724 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:18:10.776785 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:10.776786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:10.776795 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:10.776851 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:10.777130 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:10.777703 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:10.778358 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:10.778546 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:10.778590 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:10.786497 (+ 7907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50539: BlockingRecv error: recv got EOF from 127.0.98.1:50539 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":49,"threads_started":1}
Sep 25 13:18:10 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:10 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:10.893560 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:10.888891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:55927)
0925 13:18:10.889044 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:10.889055 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:10.889123 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:10.889279 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:10.889280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:10.889437 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:10.889485 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:10.889487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:10.890220 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:10.890221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:10.890758 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:10.890760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:10.891201 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:10.891202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:10.891256 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:10.891768 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:10.891772 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:10.893470 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
W20260925 13:18:10.893672 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:10.889064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55927 (local address 127.0.98.62:41827)
0925 13:18:10.889159 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:10.889161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:10.889171 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:18:10.889199 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:10.889200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:10.889212 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:10.889267 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:10.889574 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:10.890077 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:10.890849 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:10.891086 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:10.891182 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:10.893609 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55927: BlockingRecv error: recv got EOF from 127.0.98.2:55927 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 13:18:11 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:11 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:11.034852 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:11.030415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:41059)
0925 13:18:11.030576 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:11.030586 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:11.030646 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:11.030796 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:11.030796 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:11.030949 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:11.030996 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:11.030997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:11.031624 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:11.031625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:11.032261 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:11.032265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:11.032628 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:11.032629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:11.032673 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:11.033126 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:11.033131 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:11.034786 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
W20260925 13:18:11.034968 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:11.030490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41059 (local address 127.0.98.62:41827)
0925 13:18:11.030546 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:11.030548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:11.030642 (+ 94us) server_negotiation.cc:407] Connection header received
0925 13:18:11.030724 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:11.030726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:11.030738 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:11.030783 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:11.031048 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:11.031544 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:11.032332 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:11.032561 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:11.032626 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:11.034895 (+ 2269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41059: BlockingRecv error: recv got EOF from 127.0.98.3:41059 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 25 13:18:11 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:11 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:11.791146 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:11.786988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:44933)
0925 13:18:11.787129 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:11.787140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:11.787186 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:11.787333 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:11.787334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:11.787494 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:11.787542 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:11.787544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:11.788150 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:11.788151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:11.788662 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:11.788664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:11.788975 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:11.788976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:11.789011 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:11.789423 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:11.789427 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:11.791088 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
W20260925 13:18:11.791292 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:11.787100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44933 (local address 127.0.98.62:41827)
0925 13:18:11.787213 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:11.787215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:11.787223 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:18:11.787246 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:11.787246 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:11.787257 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:11.787324 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:11.787600 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:11.788093 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:11.788728 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:11.788921 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:11.788970 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:11.791218 (+ 2248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44933: BlockingRecv error: recv got EOF from 127.0.98.1:44933 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":50,"threads_started":1}
Sep 25 13:18:11 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:11 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:11.898523 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:11.894107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:52959)
0925 13:18:11.894259 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:11.894270 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:11.894339 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:11.894463 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:11.894463 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:11.894620 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:11.894679 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:11.894680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:11.895296 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:11.895297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:11.895827 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:11.895829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:11.896159 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:11.896161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:11.896216 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:11.896655 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:11.896659 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:11.898459 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
W20260925 13:18:11.898690 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:11.894219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52959 (local address 127.0.98.62:41827)
0925 13:18:11.894274 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:11.894276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:11.894334 (+ 58us) server_negotiation.cc:407] Connection header received
0925 13:18:11.894398 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:11.894399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:11.894410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:11.894448 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:11.894729 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:11.895221 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:11.895904 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:11.896080 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:11.896121 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:11.898607 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52959: BlockingRecv error: recv got EOF from 127.0.98.2:52959 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 13:18:12 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:12 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:12.039515 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:12.035297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.3:40113)
0925 13:18:12.035414 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:12.035425 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:12.035479 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:12.035608 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:12.035608 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:12.035766 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:12.035814 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:12.035815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:12.036407 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:12.036408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:12.036917 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:12.036919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:12.037204 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:12.037205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:12.037240 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:12.037697 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:12.037702 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:12.039452 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
W20260925 13:18:12.039601 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:12.035380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40113 (local address 127.0.98.62:41827)
0925 13:18:12.035442 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:12.035443 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:12.035466 (+ 23us) server_negotiation.cc:407] Connection header received
0925 13:18:12.035528 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:12.035529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:12.035540 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:12.035594 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:12.035862 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:12.036350 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:12.036978 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:12.037154 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:12.037201 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:12.039539 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40113: BlockingRecv error: recv got EOF from 127.0.98.3:40113 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 13:18:12 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:12 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:12.796061 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:12.791716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.1:59907)
0925 13:18:12.791834 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:12.791846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:12.791941 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:12.792077 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:12.792078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:12.792240 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:12.792293 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:12.792295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:12.792928 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:12.792929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:12.793587 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:12.793591 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:12.793953 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:12.793954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:12.793996 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:12.794435 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:12.794439 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:12.796000 (+ 1561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
W20260925 13:18:12.796170 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:12.791810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59907 (local address 127.0.98.62:41827)
0925 13:18:12.791955 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:12.791957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:12.791968 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:18:12.791993 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:12.791994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:12.792005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:12.792061 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:12.792354 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:12.792851 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:12.793656 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:12.793891 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:12.793948 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:12.796101 (+ 2153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59907: BlockingRecv error: recv got EOF from 127.0.98.1:59907 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
Sep 25 13:18:12 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 13:18:12 dist-test-slave-t8d9 krb5kdc[8525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 13:18:12.903997 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:12.899150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41827 (local address 127.0.98.2:33481)
0925 13:18:12.899292 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:12.899303 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:12.899366 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:12.899538 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:12.899539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:12.899693 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:12.899742 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:12.899744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:12.900359 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:12.900360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:12.900900 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:12.900903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:12.901243 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:12.901245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:12.901303 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:12.901864 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:12.901869 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:12.903892 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41827: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
W20260925 13:18:12.904076 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:12.899305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33481 (local address 127.0.98.62:41827)
0925 13:18:12.899405 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:12.899407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:12.899419 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:18:12.899449 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:12.899450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:12.899461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:12.899528 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:12.899803 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:12.900295 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:12.900970 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:12.901147 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:12.901193 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:12.904021 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33481: BlockingRecv error: recv got EOF from 127.0.98.2:33481 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 13:18:12.956884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 9108
I20260925 13:18:12.961599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 9260
I20260925 13:18:12.966320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 9392
I20260925 13:18:12.970281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 9038
2026-09-25T13:18:12Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17318 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T13:18:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:18:12Z Disabled control of system clock
I20260925 13:18:12.996373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39259
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 13:18:13.076246 9588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:18:13.076454 9588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:18:13.076474 9588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:18:13.077636 9588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:18:13.077674 9588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:18:13.077687 9588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:18:13.077697 9588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:18:13.077708 9588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:18:13.079003 9588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39259
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:18:13.079211 9588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:18:13.079460 9588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:18:13.084143 9594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:13.084208 9588 server_base.cc:1034] running on GCE node
W20260925 13:18:13.084157 9596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:13.084151 9593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:13.084673 9588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:18:13.084971 9588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:18:13.086119 9588 hybrid_clock.cc:648] HybridClock initialized: now 1790342293086102 us; error 30 us; skew 500 ppm
I20260925 13:18:13.087340 9588 webserver.cc:460] Webserver started at http://127.0.98.62:43881/ using document root <none> and password file <none>
I20260925 13:18:13.087546 9588 fs_manager.cc:359] Metadata directory not provided
I20260925 13:18:13.087594 9588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:18:13.087715 9588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:18:13.088769 9588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "ff7242707f844a2fbcfd350fe758d24b"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "6c2c33aa5c231136cc2c789e1ec985bb"
server_key_iv: "6afcc24e8283952fb54bc147455c250d"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.089103 9588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "ff7242707f844a2fbcfd350fe758d24b"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "6c2c33aa5c231136cc2c789e1ec985bb"
server_key_iv: "6afcc24e8283952fb54bc147455c250d"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.090425 9588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:18:13.091076 9602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.091264 9588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:18:13.091348 9588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "ff7242707f844a2fbcfd350fe758d24b"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "6c2c33aa5c231136cc2c789e1ec985bb"
server_key_iv: "6afcc24e8283952fb54bc147455c250d"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.091409 9588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:18:13.100553 9588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:18:13.100854 9588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:18:13.100986 9588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:18:13.104663 9588 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39259
I20260925 13:18:13.104696 9654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39259 every 8 connection(s)
I20260925 13:18:13.105029 9588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:18:13.105696 9655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:18:13.108282 9655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b: Bootstrap starting.
I20260925 13:18:13.108848 9655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b: Neither blocks nor log segments found. Creating new log.
I20260925 13:18:13.109107 9655 log.cc:824] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b: Log is configured to *not* fsync() on all Append() calls
I20260925 13:18:13.109843 9655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b: No bootstrap required, opened a new log
I20260925 13:18:13.110459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 9588
I20260925 13:18:13.110539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:18:13.110692 392 external_mini_cluster.cc:1428] Setting key 4606198076093b1ce60652b434e3af91
I20260925 13:18:13.111755 9655 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7242707f844a2fbcfd350fe758d24b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39259 } }
I20260925 13:18:13.111922 9655 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:18:13.111956 9655 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff7242707f844a2fbcfd350fe758d24b, State: Initialized, Role: FOLLOWER
I20260925 13:18:13.112112 9655 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7242707f844a2fbcfd350fe758d24b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39259 } }
I20260925 13:18:13.112188 9655 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:18:13.112210 9655 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:18:13.112236 9655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:18:13.112753 9655 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7242707f844a2fbcfd350fe758d24b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39259 } }
I20260925 13:18:13.112836 9655 leader_election.cc:302] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff7242707f844a2fbcfd350fe758d24b; no voters:
I20260925 13:18:13.112982 9655 leader_election.cc:288] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:18:13.113006 9660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:18:13.113135 9660 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [term 1 LEADER]: Becoming Leader. State: Replica: ff7242707f844a2fbcfd350fe758d24b, State: Running, Role: LEADER
I20260925 13:18:13.113237 9655 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:18:13.113273 9660 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7242707f844a2fbcfd350fe758d24b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39259 } }
I20260925 13:18:13.113698 9660 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff7242707f844a2fbcfd350fe758d24b. Latest consensus state: current_term: 1 leader_uuid: "ff7242707f844a2fbcfd350fe758d24b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7242707f844a2fbcfd350fe758d24b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39259 } } }
I20260925 13:18:13.113803 9660 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [sys.catalog]: This master's current role is: LEADER
I20260925 13:18:13.114022 9669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:18:13.114655 9662 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff7242707f844a2fbcfd350fe758d24b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7242707f844a2fbcfd350fe758d24b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39259 } } }
I20260925 13:18:13.114761 9669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:18:13.114807 9662 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b [sys.catalog]: This master's current role is: LEADER
I20260925 13:18:13.116559 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:13.111238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.0.98.62:39259)
0925 13:18:13.111489 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:13.111499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:13.111535 (+ 36us) server_negotiation.cc:407] Connection header received
0925 13:18:13.111711 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:13.111719 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:13.111791 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:13.111948 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:13.112155 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:13.113595 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:13.114619 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:13.115137 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:13.115259 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:13.115920 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:18:13.115925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:18:13.115925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:18:13.116088 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:18:13.116122 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:18:13.116133 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:18:13.116195 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:13.116282 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 13:18:13.116365 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260925 13:18:13.117008 9669 catalog_manager.cc:1347] Generated new cluster ID: fbf36f2522c341fd9489ef42bbcc177b
I20260925 13:18:13.117058 9669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:18:13.124782 9669 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:18:13.125516 9669 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:18:13.135638 9669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff7242707f844a2fbcfd350fe758d24b: Generated new TSK 0
I20260925 13:18:13.135854 9669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:18:13.142913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39259
--builtin_ntp_servers=127.0.98.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 13:18:13.223241 9679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:18:13.223474 9679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:18:13.223495 9679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:18:13.224767 9679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:18:13.224815 9679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:18:13.224851 9679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:18:13.226380 9679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:18:13.226652 9679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:18:13.226977 9679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:18:13.231961 9687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:13.232043 9684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:13.232268 9679 server_base.cc:1034] running on GCE node
W20260925 13:18:13.233966 9685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:13.234274 9679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:18:13.234606 9679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:18:13.235775 9679 hybrid_clock.cc:648] HybridClock initialized: now 1790342293235738 us; error 51 us; skew 500 ppm
I20260925 13:18:13.237365 9679 webserver.cc:460] Webserver started at http://127.0.98.1:34365/ using document root <none> and password file <none>
I20260925 13:18:13.237663 9679 fs_manager.cc:359] Metadata directory not provided
I20260925 13:18:13.237725 9679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:18:13.237874 9679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:18:13.239277 9679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "abeceb9caee749fd847ac3b974dd4861"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "b1b8bb880a2cb2ba4832e61bb7942373"
server_key_iv: "5a7206c9fe9cc9b9c4878c0a2a87a477"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.239656 9679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "abeceb9caee749fd847ac3b974dd4861"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "b1b8bb880a2cb2ba4832e61bb7942373"
server_key_iv: "5a7206c9fe9cc9b9c4878c0a2a87a477"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.241163 9679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:18:13.242110 9693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.242271 9679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:18:13.242358 9679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "abeceb9caee749fd847ac3b974dd4861"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "b1b8bb880a2cb2ba4832e61bb7942373"
server_key_iv: "5a7206c9fe9cc9b9c4878c0a2a87a477"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.242432 9679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:18:13.291476 9679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:18:13.291769 9679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:18:13.291879 9679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:18:13.292069 9679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:18:13.292354 9679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:18:13.292384 9679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.292404 9679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:18:13.292420 9679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.298089 9679 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41317
I20260925 13:18:13.298148 9806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41317 every 8 connection(s)
I20260925 13:18:13.298458 9679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:18:13.303402 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:13.299122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47517 (local address 127.0.98.62:39259)
0925 13:18:13.299174 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:13.299176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:13.299433 (+ 257us) server_negotiation.cc:407] Connection header received
0925 13:18:13.299918 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:13.299919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:13.299931 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:13.299990 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:13.300645 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:13.301134 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:13.302135 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:13.302321 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:13.302369 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:13.303101 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:18:13.303103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:18:13.303103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:18:13.303108 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:18:13.303121 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:18:13.303125 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:18:13.303164 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:13.303317 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 13:18:13.303340 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:18:13.303640 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:13.299111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39259 (local address 127.0.98.1:47517)
0925 13:18:13.299306 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:13.299348 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:13.299832 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:13.300076 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:13.300080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:13.300089 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:13.300581 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:13.300584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:13.301203 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:13.301204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:13.302066 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:13.302068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:13.302395 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:13.302395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:13.302506 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:13.302973 (+ 467us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:18:13.302973 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:13.303003 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:18:13.303026 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:18:13.303211 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:18:13.303218 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:18:13.303221 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:18:13.303317 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 13:18:13.303418 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260925 13:18:13.304102 9807 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39259
I20260925 13:18:13.304186 9807 heartbeater.cc:464] Registering TS with master...
I20260925 13:18:13.304361 9807 heartbeater.cc:511] Master 127.0.98.62:39259 requested a full tablet report, sending...
I20260925 13:18:13.304808 9619 ts_manager.cc:194] Registered new tserver with Master: abeceb9caee749fd847ac3b974dd4861 (127.0.98.1:41317)
I20260925 13:18:13.305403 9619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47517
I20260925 13:18:13.308099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 9679
I20260925 13:18:13.308175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:18:13.308352 392 external_mini_cluster.cc:1428] Setting key 9b9291a2200698906218cc319dbe0959
I20260925 13:18:13.309197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39259
--builtin_ntp_servers=127.0.98.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 13:18:13.389348 9810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:18:13.389586 9810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:18:13.389617 9810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:18:13.390810 9810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:18:13.390854 9810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:18:13.390892 9810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:18:13.392246 9810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:18:13.392491 9810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:18:13.392776 9810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:18:13.397710 9815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:13.397711 9818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:13.397753 9816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:13.397763 9810 server_base.cc:1034] running on GCE node
I20260925 13:18:13.398151 9810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:18:13.398362 9810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:18:13.399500 9810 hybrid_clock.cc:648] HybridClock initialized: now 1790342293399486 us; error 33 us; skew 500 ppm
I20260925 13:18:13.400712 9810 webserver.cc:460] Webserver started at http://127.0.98.2:33061/ using document root <none> and password file <none>
I20260925 13:18:13.400945 9810 fs_manager.cc:359] Metadata directory not provided
I20260925 13:18:13.400991 9810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:18:13.401116 9810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:18:13.402254 9810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "5fcb87667f9e4f5aa59984dac592465c"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "05cd61920e8dd303a46e98e08c7f51d9"
server_key_iv: "3ca5b2df9b5187cf7b9f6b3f75ef58bc"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.402593 9810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5fcb87667f9e4f5aa59984dac592465c"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "05cd61920e8dd303a46e98e08c7f51d9"
server_key_iv: "3ca5b2df9b5187cf7b9f6b3f75ef58bc"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.403769 9810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:18:13.404456 9824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.404637 9810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:18:13.404713 9810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "5fcb87667f9e4f5aa59984dac592465c"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "05cd61920e8dd303a46e98e08c7f51d9"
server_key_iv: "3ca5b2df9b5187cf7b9f6b3f75ef58bc"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.404774 9810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:18:13.414646 9810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:18:13.414937 9810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:18:13.415055 9810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:18:13.415258 9810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:18:13.415589 9810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:18:13.415624 9810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.415652 9810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:18:13.415674 9810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.421095 9810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45221
I20260925 13:18:13.421145 9937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45221 every 8 connection(s)
I20260925 13:18:13.421492 9810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:18:13.423611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 9810
I20260925 13:18:13.423691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:18:13.423854 392 external_mini_cluster.cc:1428] Setting key 2fe74bb824a7f9298e44b2caa6557bf3
I20260925 13:18:13.424732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39259
--builtin_ntp_servers=127.0.98.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 13:18:13.427011 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:13.422312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55979 (local address 127.0.98.62:39259)
0925 13:18:13.422363 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:13.422366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:13.422623 (+ 257us) server_negotiation.cc:407] Connection header received
0925 13:18:13.423106 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:13.423107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:13.423118 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:13.423165 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:13.423882 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:13.424433 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:13.425467 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:13.425766 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:13.425842 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:13.426745 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:18:13.426747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:18:13.426747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:18:13.426751 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:18:13.426765 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:18:13.426769 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:18:13.426809 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:13.426905 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 13:18:13.426935 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:18:13.427201 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:13.422307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39259 (local address 127.0.98.2:55979)
0925 13:18:13.422500 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:13.422536 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:13.423026 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:13.423229 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:13.423233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:13.423241 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:13.423789 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:13.423797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:13.424509 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:13.424510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:13.425380 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:13.425383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:13.425863 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:13.425864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:13.426036 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:13.426580 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:18:13.426581 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:13.426622 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:18:13.426657 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:18:13.426820 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:18:13.426825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:18:13.426828 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:18:13.426887 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 13:18:13.427003 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260925 13:18:13.427721 9938 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39259
I20260925 13:18:13.427805 9938 heartbeater.cc:464] Registering TS with master...
I20260925 13:18:13.427974 9938 heartbeater.cc:511] Master 127.0.98.62:39259 requested a full tablet report, sending...
I20260925 13:18:13.428313 9619 ts_manager.cc:194] Registered new tserver with Master: 5fcb87667f9e4f5aa59984dac592465c (127.0.98.2:45221)
I20260925 13:18:13.428671 9619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55979
W20260925 13:18:13.505046 9941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:18:13.505263 9941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:18:13.505285 9941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:18:13.506507 9941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:18:13.506552 9941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:18:13.506577 9941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:18:13.507998 9941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:18:13.508229 9941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:18:13.508569 9941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:18:13.513231 9946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:13.513242 9947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:13.513240 9949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:13.513454 9941 server_base.cc:1034] running on GCE node
I20260925 13:18:13.513724 9941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:18:13.513969 9941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:18:13.515111 9941 hybrid_clock.cc:648] HybridClock initialized: now 1790342293515095 us; error 32 us; skew 500 ppm
I20260925 13:18:13.516325 9941 webserver.cc:460] Webserver started at http://127.0.98.3:40457/ using document root <none> and password file <none>
I20260925 13:18:13.516564 9941 fs_manager.cc:359] Metadata directory not provided
I20260925 13:18:13.516610 9941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:18:13.516741 9941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:18:13.517903 9941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "6ef05dfff3374209b98961f515754faa"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "ad23373958c6c485c0a131a063700b1b"
server_key_iv: "2b3cd5422c6a86ee80206f521af2ebb1"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.518252 9941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6ef05dfff3374209b98961f515754faa"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "ad23373958c6c485c0a131a063700b1b"
server_key_iv: "2b3cd5422c6a86ee80206f521af2ebb1"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.519484 9941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:18:13.520153 9955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.520300 9941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:18:13.520377 9941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "6ef05dfff3374209b98961f515754faa"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "ad23373958c6c485c0a131a063700b1b"
server_key_iv: "2b3cd5422c6a86ee80206f521af2ebb1"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.520447 9941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:18:13.528326 9941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:18:13.528599 9941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:18:13.528708 9941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:18:13.528916 9941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:18:13.529227 9941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:18:13.529259 9941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.529280 9941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:18:13.529302 9941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.534673 9941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38965
I20260925 13:18:13.534737 10068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38965 every 8 connection(s)
I20260925 13:18:13.535060 9941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:18:13.539811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 9941
I20260925 13:18:13.539906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:18:13.540017 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:13.535740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43335 (local address 127.0.98.62:39259)
0925 13:18:13.535788 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:13.535790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:13.536018 (+ 228us) server_negotiation.cc:407] Connection header received
0925 13:18:13.536525 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:13.536526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:13.536539 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:13.536590 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:13.537247 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:13.537799 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:13.538745 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:13.538938 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:13.538993 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:13.539744 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:18:13.539746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:18:13.539746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:18:13.539751 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:18:13.539766 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:18:13.539770 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:18:13.539810 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:13.539903 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 13:18:13.539933 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:18:13.540108 392 external_mini_cluster.cc:1428] Setting key 87091d1372eceeafea8b1b8a495a2131
I20260925 13:18:13.540238 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:13.535728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39259 (local address 127.0.98.3:43335)
0925 13:18:13.535893 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:13.535933 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:13.536430 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:13.536662 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:13.536666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:13.536676 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:13.537164 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:13.537168 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:13.537881 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:13.537882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:13.538665 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:13.538667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:13.539009 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:13.539009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:13.539128 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:13.539589 (+ 461us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:18:13.539590 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:13.539619 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:18:13.539648 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:18:13.539823 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:18:13.539828 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:18:13.539831 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:18:13.539914 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 13:18:13.540025 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260925 13:18:13.540717 10069 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39259
I20260925 13:18:13.540812 10069 heartbeater.cc:464] Registering TS with master...
I20260925 13:18:13.541008 10069 heartbeater.cc:511] Master 127.0.98.62:39259 requested a full tablet report, sending...
I20260925 13:18:13.541337 9619 ts_manager.cc:194] Registered new tserver with Master: 6ef05dfff3374209b98961f515754faa (127.0.98.3:38965)
I20260925 13:18:13.541734 9619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:43335
I20260925 13:18:13.551754 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 13:18:13.553551 9659 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:13.553023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47750 (local address 127.0.98.62:39259)
0925 13:18:13.553082 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:13.553084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:13.553174 (+ 90us) server_negotiation.cc:407] Connection header received
0925 13:18:13.553239 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:13.553240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:13.553253 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:13.553298 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:13.553456 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47750: BlockingRecv error: recv got EOF from 127.0.0.1:47750 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 13:18:13.553574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 9679
I20260925 13:18:13.557994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 9810
I20260925 13:18:13.562057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 9941
I20260925 13:18:13.565830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 9588
2026-09-25T13:18:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (594 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T13:18:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:18:13Z Disabled control of system clock
I20260925 13:18:13.591096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44383
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 13:18:13.669565 10080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:18:13.669788 10080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:18:13.669816 10080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:18:13.671015 10080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:18:13.671059 10080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:18:13.671074 10080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:18:13.671090 10080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:18:13.671105 10080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:18:13.672477 10080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44383
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:18:13.672729 10080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:18:13.673022 10080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:18:13.677647 10085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:13.677672 10088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:13.677680 10086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:13.677709 10080 server_base.cc:1034] running on GCE node
I20260925 13:18:13.678041 10080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:18:13.678318 10080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:18:13.679464 10080 hybrid_clock.cc:648] HybridClock initialized: now 1790342293679448 us; error 29 us; skew 500 ppm
I20260925 13:18:13.680634 10080 webserver.cc:460] Webserver started at http://127.0.98.62:33853/ using document root <none> and password file <none>
I20260925 13:18:13.680851 10080 fs_manager.cc:359] Metadata directory not provided
I20260925 13:18:13.680899 10080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:18:13.681039 10080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:18:13.682147 10080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "6d19e5bab73a433a875ee2a265915a2a"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "2a738961bde5bd757072a4750c175acf"
server_key_iv: "8415f7e375386c140d052fa70695a9e6"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.682475 10080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "6d19e5bab73a433a875ee2a265915a2a"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "2a738961bde5bd757072a4750c175acf"
server_key_iv: "8415f7e375386c140d052fa70695a9e6"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.683698 10080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:18:13.684419 10094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.684574 10080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:18:13.684666 10080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "6d19e5bab73a433a875ee2a265915a2a"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "2a738961bde5bd757072a4750c175acf"
server_key_iv: "8415f7e375386c140d052fa70695a9e6"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.684723 10080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:18:13.687963 10080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:18:13.688192 10080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:18:13.688283 10080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:18:13.691812 10080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44383
I20260925 13:18:13.691843 10146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44383 every 8 connection(s)
I20260925 13:18:13.692157 10080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:18:13.692763 10147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:18:13.694856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 10080
I20260925 13:18:13.694934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:18:13.695120 392 external_mini_cluster.cc:1428] Setting key 0059a34b97cf975f5a588e5f263d70e5
I20260925 13:18:13.695729 10147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a: Bootstrap starting.
I20260925 13:18:13.696329 10147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a: Neither blocks nor log segments found. Creating new log.
I20260925 13:18:13.696600 10147 log.cc:824] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a: Log is configured to *not* fsync() on all Append() calls
I20260925 13:18:13.697338 10147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a: No bootstrap required, opened a new log
I20260925 13:18:13.697579 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:13.695621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.0.98.62:44383)
0925 13:18:13.695788 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:13.695796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:13.695825 (+ 29us) server_negotiation.cc:407] Connection header received
0925 13:18:13.696253 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:13.696261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:13.696303 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:13.696436 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:13.697081 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:18:13.697085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:18:13.697085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:18:13.697090 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:18:13.697141 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:18:13.697155 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:18:13.697249 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:13.697281 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 13:18:13.697336 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260925 13:18:13.698927 10147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d19e5bab73a433a875ee2a265915a2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44383 } }
I20260925 13:18:13.699069 10147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:18:13.699090 10147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d19e5bab73a433a875ee2a265915a2a, State: Initialized, Role: FOLLOWER
I20260925 13:18:13.699180 10147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d19e5bab73a433a875ee2a265915a2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44383 } }
I20260925 13:18:13.699508 10147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:18:13.699539 10147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:18:13.699571 10147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:18:13.700062 10147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d19e5bab73a433a875ee2a265915a2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44383 } }
I20260925 13:18:13.700145 10147 leader_election.cc:302] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d19e5bab73a433a875ee2a265915a2a; no voters:
I20260925 13:18:13.700264 10147 leader_election.cc:288] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:18:13.700320 10152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:18:13.700495 10147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:18:13.700481 10152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [term 1 LEADER]: Becoming Leader. State: Replica: 6d19e5bab73a433a875ee2a265915a2a, State: Running, Role: LEADER
I20260925 13:18:13.700628 10152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d19e5bab73a433a875ee2a265915a2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44383 } }
I20260925 13:18:13.700986 10154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d19e5bab73a433a875ee2a265915a2a. Latest consensus state: current_term: 1 leader_uuid: "6d19e5bab73a433a875ee2a265915a2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d19e5bab73a433a875ee2a265915a2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44383 } } }
I20260925 13:18:13.700989 10153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d19e5bab73a433a875ee2a265915a2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d19e5bab73a433a875ee2a265915a2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44383 } } }
I20260925 13:18:13.701120 10154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [sys.catalog]: This master's current role is: LEADER
I20260925 13:18:13.701126 10153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a [sys.catalog]: This master's current role is: LEADER
I20260925 13:18:13.701455 10161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:18:13.701954 10161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:18:13.703466 10161 catalog_manager.cc:1347] Generated new cluster ID: 20c6416555ba4f69bda50c03dff01e81
I20260925 13:18:13.703509 10161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:18:13.712476 10161 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:18:13.712639 10161 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:18:13.714346 10161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d19e5bab73a433a875ee2a265915a2a: Generated new TSK 0
I20260925 13:18:13.714517 10161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:18:13.718153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44383
--builtin_ntp_servers=127.0.98.20:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 13:18:13.799682 10171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:18:13.799909 10171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:18:13.799948 10171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:18:13.801275 10171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:18:13.801316 10171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:18:13.801352 10171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:18:13.802872 10171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:18:13.803130 10171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:18:13.803426 10171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:18:13.808240 10176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:13.808226 10177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:13.808220 10179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:13.808617 10171 server_base.cc:1034] running on GCE node
I20260925 13:18:13.808807 10171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:18:13.809053 10171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:18:13.810196 10171 hybrid_clock.cc:648] HybridClock initialized: now 1790342293810182 us; error 32 us; skew 500 ppm
I20260925 13:18:13.811504 10171 webserver.cc:460] Webserver started at http://127.0.98.1:37921/ using document root <none> and password file <none>
I20260925 13:18:13.811743 10171 fs_manager.cc:359] Metadata directory not provided
I20260925 13:18:13.811791 10171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:18:13.811921 10171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:18:13.812968 10171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "a56684fed404467dae5bcb14da96f749"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "9bc77316bdb0620e5a0428ce6ec6e4a6"
server_key_iv: "1bd157ef9be259d1dc69e395baf914d0"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.813304 10171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a56684fed404467dae5bcb14da96f749"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "9bc77316bdb0620e5a0428ce6ec6e4a6"
server_key_iv: "1bd157ef9be259d1dc69e395baf914d0"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.814630 10171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:18:13.815346 10185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.815492 10171 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:18:13.815593 10171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "a56684fed404467dae5bcb14da96f749"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "9bc77316bdb0620e5a0428ce6ec6e4a6"
server_key_iv: "1bd157ef9be259d1dc69e395baf914d0"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.815661 10171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:18:13.820360 10171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:18:13.820668 10171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:18:13.820798 10171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:18:13.821017 10171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:18:13.821352 10171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:18:13.821396 10171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.821422 10171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:18:13.821439 10171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.828405 10171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38645
I20260925 13:18:13.828467 10298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38645 every 8 connection(s)
I20260925 13:18:13.828832 10171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:18:13.832103 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:13.829676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57101 (local address 127.0.98.62:44383)
0925 13:18:13.829744 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:13.829747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:13.830039 (+ 292us) server_negotiation.cc:407] Connection header received
0925 13:18:13.830520 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:13.830522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:13.830528 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:13.830610 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:13.831800 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:18:13.831802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:18:13.831802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:18:13.831807 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:18:13.831824 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:18:13.831829 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:18:13.831858 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:13.831991 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 13:18:13.832029 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:18:13.832216 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:13.829700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44383 (local address 127.0.98.1:57101)
0925 13:18:13.829894 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:13.829937 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:13.830415 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:13.830669 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:13.830673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:13.830682 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:13.831611 (+ 929us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:18:13.831612 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:13.831643 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:18:13.831697 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:18:13.831899 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:18:13.831905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:18:13.831908 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:18:13.831970 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 13:18:13.832027 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":104,"threads_started":1}
I20260925 13:18:13.832691 10299 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44383
I20260925 13:18:13.832803 10299 heartbeater.cc:464] Registering TS with master...
I20260925 13:18:13.832968 10299 heartbeater.cc:511] Master 127.0.98.62:44383 requested a full tablet report, sending...
I20260925 13:18:13.833482 10111 ts_manager.cc:194] Registered new tserver with Master: a56684fed404467dae5bcb14da96f749 (127.0.98.1:38645)
I20260925 13:18:13.834178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 10171
I20260925 13:18:13.834266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:18:13.834438 392 external_mini_cluster.cc:1428] Setting key b1ed593c979a4824702e02e444ecce8c
I20260925 13:18:13.835242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44383
--builtin_ntp_servers=127.0.98.20:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 13:18:13.915364 10302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:18:13.915583 10302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:18:13.915606 10302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:18:13.916760 10302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:18:13.916798 10302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:18:13.916832 10302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:18:13.918200 10302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:18:13.918409 10302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:18:13.918663 10302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:18:13.923573 10310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:13.923606 10308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:13.923580 10307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:13.923806 10302 server_base.cc:1034] running on GCE node
I20260925 13:18:13.924094 10302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:18:13.924322 10302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:18:13.925462 10302 hybrid_clock.cc:648] HybridClock initialized: now 1790342293925447 us; error 29 us; skew 500 ppm
I20260925 13:18:13.926656 10302 webserver.cc:460] Webserver started at http://127.0.98.2:34929/ using document root <none> and password file <none>
I20260925 13:18:13.926884 10302 fs_manager.cc:359] Metadata directory not provided
I20260925 13:18:13.926932 10302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:18:13.927054 10302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:18:13.928085 10302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "a57aedb2caff4be1a0d7167d037eb817"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "7f0c63b287797f4ccf8f3c2812ecea01"
server_key_iv: "c167dae1aa70859638f4d54351de37d0"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.928414 10302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a57aedb2caff4be1a0d7167d037eb817"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "7f0c63b287797f4ccf8f3c2812ecea01"
server_key_iv: "c167dae1aa70859638f4d54351de37d0"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.929664 10302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:18:13.930362 10316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.930529 10302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.930605 10302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "a57aedb2caff4be1a0d7167d037eb817"
format_stamp: "Formatted at 2026-09-25 13:18:13 on dist-test-slave-t8d9"
server_key: "7f0c63b287797f4ccf8f3c2812ecea01"
server_key_iv: "c167dae1aa70859638f4d54351de37d0"
server_key_version: "encryptionkey@0"
I20260925 13:18:13.930670 10302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:18:13.934011 10302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:18:13.934269 10302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:18:13.934396 10302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:18:13.934573 10302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:18:13.934870 10302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:18:13.934907 10302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.934934 10302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:18:13.934957 10302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:13.940398 10302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36049
I20260925 13:18:13.940435 10429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36049 every 8 connection(s)
I20260925 13:18:13.940739 10302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:18:13.943432 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:13.941392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34467 (local address 127.0.98.62:44383)
0925 13:18:13.941438 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:13.941440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:13.941689 (+ 249us) server_negotiation.cc:407] Connection header received
0925 13:18:13.942144 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:13.942145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:13.942150 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:13.942199 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:13.943250 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:18:13.943251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:18:13.943252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:18:13.943255 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:18:13.943268 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:18:13.943272 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:18:13.943309 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:13.943364 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 13:18:13.943374 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:18:13.943611 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:13.941384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44383 (local address 127.0.98.2:34467)
0925 13:18:13.941587 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:13.941622 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:13.942067 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:13.942253 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:13.942256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:13.942264 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:13.943111 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:18:13.943111 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:13.943140 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:18:13.943170 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:18:13.943313 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:18:13.943316 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:18:13.943318 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:18:13.943366 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 13:18:13.943412 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260925 13:18:13.944041 10430 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44383
I20260925 13:18:13.944126 10430 heartbeater.cc:464] Registering TS with master...
I20260925 13:18:13.944238 10430 heartbeater.cc:511] Master 127.0.98.62:44383 requested a full tablet report, sending...
I20260925 13:18:13.944545 10111 ts_manager.cc:194] Registered new tserver with Master: a57aedb2caff4be1a0d7167d037eb817 (127.0.98.2:36049)
I20260925 13:18:13.949819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 10302
I20260925 13:18:13.949903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:18:13.950208 392 external_mini_cluster.cc:1428] Setting key 55264998ad535566e5a5160238c6c02b
I20260925 13:18:13.951205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44383
--builtin_ntp_servers=127.0.98.20:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 13:18:14.031194 10433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:18:14.031417 10433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:18:14.031441 10433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:18:14.032670 10433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:18:14.032711 10433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:18:14.032737 10433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:18:14.034199 10433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:18:14.034453 10433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:18:14.034737 10433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:18:14.039609 10438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:14.039618 10439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:14.039711 10433 server_base.cc:1034] running on GCE node
W20260925 13:18:14.039618 10441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:14.039996 10433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:18:14.040254 10433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:18:14.041401 10433 hybrid_clock.cc:648] HybridClock initialized: now 1790342294041388 us; error 38 us; skew 500 ppm
I20260925 13:18:14.042677 10433 webserver.cc:460] Webserver started at http://127.0.98.3:33475/ using document root <none> and password file <none>
I20260925 13:18:14.042924 10433 fs_manager.cc:359] Metadata directory not provided
I20260925 13:18:14.042979 10433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:18:14.043107 10433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:18:14.044209 10433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "8f6209f0ab7f42b28d57de08ceb469a9"
format_stamp: "Formatted at 2026-09-25 13:18:14 on dist-test-slave-t8d9"
server_key: "83f8f556211308fb8469686458be0395"
server_key_iv: "9a9a09a818d65ab156f0655453f0960b"
server_key_version: "encryptionkey@0"
I20260925 13:18:14.044555 10433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8f6209f0ab7f42b28d57de08ceb469a9"
format_stamp: "Formatted at 2026-09-25 13:18:14 on dist-test-slave-t8d9"
server_key: "83f8f556211308fb8469686458be0395"
server_key_iv: "9a9a09a818d65ab156f0655453f0960b"
server_key_version: "encryptionkey@0"
I20260925 13:18:14.045840 10433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:18:14.046576 10447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:14.046715 10433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:18:14.046798 10433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "8f6209f0ab7f42b28d57de08ceb469a9"
format_stamp: "Formatted at 2026-09-25 13:18:14 on dist-test-slave-t8d9"
server_key: "83f8f556211308fb8469686458be0395"
server_key_iv: "9a9a09a818d65ab156f0655453f0960b"
server_key_version: "encryptionkey@0"
I20260925 13:18:14.046867 10433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:18:14.050419 10433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:18:14.050722 10433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:18:14.050853 10433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:18:14.051050 10433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:18:14.051354 10433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:18:14.051386 10433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:14.051414 10433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:18:14.051447 10433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:14.056840 10433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45697
I20260925 13:18:14.056885 10560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45697 every 8 connection(s)
I20260925 13:18:14.057210 10433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:18:14.060154 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:14.057925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54131 (local address 127.0.98.62:44383)
0925 13:18:14.057975 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:14.057977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:14.058208 (+ 231us) server_negotiation.cc:407] Connection header received
0925 13:18:14.058706 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:14.058707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:14.058714 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:14.058782 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:14.059918 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:18:14.059919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:18:14.059920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:18:14.059924 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:18:14.059942 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:18:14.059946 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:18:14.059994 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:14.060068 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 13:18:14.060082 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:18:14.060374 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:14.057906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44383 (local address 127.0.98.3:54131)
0925 13:18:14.058089 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:14.058126 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:14.058608 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:14.058864 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:14.058867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:14.058875 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:14.059761 (+ 886us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:18:14.059762 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:14.059793 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:18:14.059824 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:18:14.059999 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:18:14.060004 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:18:14.060007 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:18:14.060075 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 13:18:14.060142 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260925 13:18:14.060899 10561 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44383
I20260925 13:18:14.061002 10561 heartbeater.cc:464] Registering TS with master...
I20260925 13:18:14.061148 10561 heartbeater.cc:511] Master 127.0.98.62:44383 requested a full tablet report, sending...
I20260925 13:18:14.061501 10111 ts_manager.cc:194] Registered new tserver with Master: 8f6209f0ab7f42b28d57de08ceb469a9 (127.0.98.3:45697)
I20260925 13:18:14.065778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 10433
I20260925 13:18:14.065853 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:18:14.066015 392 external_mini_cluster.cc:1428] Setting key a9d2df7c0b3922d1ae43424e729429bf
I20260925 13:18:14.066781 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 13:18:14.068611 10150 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:18:14.068044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.0.98.62:44383)
0925 13:18:14.068115 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:14.068117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:14.068275 (+ 158us) server_negotiation.cc:407] Connection header received
0925 13:18:14.068334 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:14.068335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:14.068342 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:14.068404 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:14.068540 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44394: BlockingRecv error: recv got EOF from 127.0.0.1:44394 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 13:18:14.069113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 10171
I20260925 13:18:14.073828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 10302
I20260925 13:18:14.077849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 10433
I20260925 13:18:14.081212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 10080
2026-09-25T13:18:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (515 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:18:14 dist-test-slave-t8d9 krb5kdc[10571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:18:14 dist-test-slave-t8d9 krb5kdc[10571](info): set up 2 sockets
Sep 25 13:18:14 dist-test-slave-t8d9 krb5kdc[10571](info): commencing operation
krb5kdc: starting...
W20260925 13:18:15.038249 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:18:15 dist-test-slave-t8d9 krb5kdc[10571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:18:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:18:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:18:15.150087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40657
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:18:15.231536 10584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:18:15.231771 10584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:18:15.231794 10584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:18:15.233045 10584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:18:15.233096 10584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:18:15.233109 10584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:18:15.233124 10584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:18:15.233146 10584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:18:15.234825 10584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:18:15.235083 10584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:18:15.235303 10584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:18:15.240150 10589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:15.240176 10592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:15.240366 10584 server_base.cc:1034] running on GCE node
W20260925 13:18:15.240152 10590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:15.240691 10584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:18:15.241006 10584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:18:15.242154 10584 hybrid_clock.cc:648] HybridClock initialized: now 1790342295242145 us; error 34 us; skew 500 ppm
Sep 25 13:18:15 dist-test-slave-t8d9 krb5kdc[10571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:18:15.244751 10584 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:18:15.245260 10584 webserver.cc:460] Webserver started at http://127.0.98.62:42911/ using document root <none> and password file <none>
I20260925 13:18:15.245482 10584 fs_manager.cc:359] Metadata directory not provided
I20260925 13:18:15.245566 10584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:18:15.245714 10584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:18:15.246810 10584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "fe3f3fa572064a9188e281f6d803288b"
format_stamp: "Formatted at 2026-09-25 13:18:15 on dist-test-slave-t8d9"
server_key: "56b0e28d958bcc2fb41b3c4a62a2f86c"
server_key_iv: "babf89998b8553c483132046bb8da83f"
server_key_version: "encryptionkey@0"
I20260925 13:18:15.247151 10584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe3f3fa572064a9188e281f6d803288b"
format_stamp: "Formatted at 2026-09-25 13:18:15 on dist-test-slave-t8d9"
server_key: "56b0e28d958bcc2fb41b3c4a62a2f86c"
server_key_iv: "babf89998b8553c483132046bb8da83f"
server_key_version: "encryptionkey@0"
I20260925 13:18:15.248423 10584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:18:15.249109 10599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:15.249248 10584 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:18:15.249333 10584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "fe3f3fa572064a9188e281f6d803288b"
format_stamp: "Formatted at 2026-09-25 13:18:15 on dist-test-slave-t8d9"
server_key: "56b0e28d958bcc2fb41b3c4a62a2f86c"
server_key_iv: "babf89998b8553c483132046bb8da83f"
server_key_version: "encryptionkey@0"
I20260925 13:18:15.249398 10584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:18:15.261354 10584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:18:15.263710 10584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:18:15.263859 10584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:18:15.267725 10584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40657
I20260925 13:18:15.267757 10651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40657 every 8 connection(s)
I20260925 13:18:15.268095 10584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:18:15.269049 10652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:18:15.271436 10652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b: Bootstrap starting.
I20260925 13:18:15.272024 10652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b: Neither blocks nor log segments found. Creating new log.
I20260925 13:18:15.272290 10652 log.cc:824] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b: Log is configured to *not* fsync() on all Append() calls
I20260925 13:18:15.273007 10652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b: No bootstrap required, opened a new log
I20260925 13:18:15.274736 10652 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3f3fa572064a9188e281f6d803288b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40657 } }
I20260925 13:18:15.274848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 10584
I20260925 13:18:15.274890 10652 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:18:15.274914 10652 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3f3fa572064a9188e281f6d803288b, State: Initialized, Role: FOLLOWER
I20260925 13:18:15.274930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:18:15.275049 10652 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3f3fa572064a9188e281f6d803288b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40657 } }
I20260925 13:18:15.275120 10652 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:18:15.275135 392 external_mini_cluster.cc:1428] Setting key 7c9ac8a7bfa1e6059e3116604888d246
I20260925 13:18:15.275148 10652 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:18:15.275189 10652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:18:15.276180 10652 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3f3fa572064a9188e281f6d803288b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40657 } }
I20260925 13:18:15.276296 10652 leader_election.cc:302] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe3f3fa572064a9188e281f6d803288b; no voters:
I20260925 13:18:15.276463 10652 leader_election.cc:288] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:18:15.276497 10657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:18:15.276659 10657 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [term 1 LEADER]: Becoming Leader. State: Replica: fe3f3fa572064a9188e281f6d803288b, State: Running, Role: LEADER
I20260925 13:18:15.276729 10652 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:18:15.276821 10657 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3f3fa572064a9188e281f6d803288b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40657 } }
I20260925 13:18:15.277158 10657 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe3f3fa572064a9188e281f6d803288b. Latest consensus state: current_term: 1 leader_uuid: "fe3f3fa572064a9188e281f6d803288b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3f3fa572064a9188e281f6d803288b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40657 } } }
I20260925 13:18:15.277170 10659 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe3f3fa572064a9188e281f6d803288b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3f3fa572064a9188e281f6d803288b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40657 } } }
I20260925 13:18:15.277330 10657 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [sys.catalog]: This master's current role is: LEADER
I20260925 13:18:15.277336 10659 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b [sys.catalog]: This master's current role is: LEADER
I20260925 13:18:15.277585 10666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:18:15.278326 10666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:18:15.280435 10666 catalog_manager.cc:1347] Generated new cluster ID: 2d7b934c5c414f4f9ad8143515be5081
I20260925 13:18:15.280493 10666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:18:15 dist-test-slave-t8d9 krb5kdc[10571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:18:15.285229 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:15.275666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54310 (local address 127.0.98.62:40657)
0925 13:18:15.275844 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:15.275852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:15.275879 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:18:15.275983 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:15.275988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:15.276035 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:15.276148 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:15.276954 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.278397 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.279322 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.279908 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.280024 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:15.282268 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:18:15.282272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:18:15.282273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:18:15.282284 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:18:15.283818 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:18:15.284148 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:18:15.284151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:18:15.284151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:18:15.284246 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:18:15.284417 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:18:15.284417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:18:15.284418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:18:15.284680 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:18:15.284818 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:15.284929 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 13:18:15.285067 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260925 13:18:15.293936 10666 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:18:15.294471 10666 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:18:15.301807 10666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b: Generated new TSK 0
I20260925 13:18:15.302057 10666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:18:15.336050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40657
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:18:15.437640 10678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:18:15.437852 10678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:18:15.437885 10678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:18:15.439148 10678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:18:15.439196 10678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:18:15.439234 10678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:18:15.440749 10678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:18:15.440991 10678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:18:15.441186 10678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:18:15.446059 10678 server_base.cc:1034] running on GCE node
W20260925 13:18:15.446030 10683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:15.446030 10686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:15.446056 10684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:15.446439 10678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:18:15.446646 10678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:18:15.447781 10678 hybrid_clock.cc:648] HybridClock initialized: now 1790342295447771 us; error 28 us; skew 500 ppm
Sep 25 13:18:15 dist-test-slave-t8d9 krb5kdc[10571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:18:15.450320 10678 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:18:15.450784 10678 webserver.cc:460] Webserver started at http://127.0.98.1:33989/ using document root <none> and password file <none>
I20260925 13:18:15.451018 10678 fs_manager.cc:359] Metadata directory not provided
I20260925 13:18:15.451067 10678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:18:15.451193 10678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:18:15.452364 10678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "1affb76bbb784605a874f035be495154"
format_stamp: "Formatted at 2026-09-25 13:18:15 on dist-test-slave-t8d9"
server_key: "5de46303762b13ed3c247d0d0658c1f1"
server_key_iv: "3b0fa91e19ec9697e84ba3f4db015397"
server_key_version: "encryptionkey@0"
I20260925 13:18:15.452723 10678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1affb76bbb784605a874f035be495154"
format_stamp: "Formatted at 2026-09-25 13:18:15 on dist-test-slave-t8d9"
server_key: "5de46303762b13ed3c247d0d0658c1f1"
server_key_iv: "3b0fa91e19ec9697e84ba3f4db015397"
server_key_version: "encryptionkey@0"
I20260925 13:18:15.454147 10678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:18:15.454865 10693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:15.455029 10678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:18:15.455111 10678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "1affb76bbb784605a874f035be495154"
format_stamp: "Formatted at 2026-09-25 13:18:15 on dist-test-slave-t8d9"
server_key: "5de46303762b13ed3c247d0d0658c1f1"
server_key_iv: "3b0fa91e19ec9697e84ba3f4db015397"
server_key_version: "encryptionkey@0"
I20260925 13:18:15.455173 10678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:18:15.473453 10678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:18:15.475732 10678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:18:15.475874 10678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:18:15.476112 10678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:18:15.476434 10678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:18:15.476475 10678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:15.476504 10678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:18:15.476567 10678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:15.482144 10678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35639
I20260925 13:18:15.482195 10806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35639 every 8 connection(s)
I20260925 13:18:15.482510 10678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 13:18:15 dist-test-slave-t8d9 krb5kdc[10571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:18:15.490949 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:15.483179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41805 (local address 127.0.98.62:40657)
0925 13:18:15.483231 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:15.483233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:15.483427 (+ 194us) server_negotiation.cc:407] Connection header received
0925 13:18:15.483919 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:15.483921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:15.483932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:15.483993 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:15.484943 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.485434 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.486383 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.486582 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.486631 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:15.488593 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:18:15.488595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:18:15.488596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:18:15.488603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:18:15.489914 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:18:15.490240 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:18:15.490241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:18:15.490242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:18:15.490285 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:18:15.490456 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:18:15.490456 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:18:15.490457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:18:15.490563 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:18:15.490608 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:15.490821 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 13:18:15.490883 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:18:15.491155 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:15.483145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40657 (local address 127.0.98.1:41805)
0925 13:18:15.483303 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:15.483346 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:15.483839 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:15.484069 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:15.484073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:15.484664 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:15.484869 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:15.484873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.485500 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.485501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:15.486317 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:15.486319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.486640 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.486641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:15.486739 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:15.487170 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:15.487178 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:15.488506 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:18:15.489980 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:18:15.489984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:18:15.489986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:18:15.490189 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:18:15.490328 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:18:15.490328 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:18:15.490328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:18:15.490416 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:18:15.490604 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:18:15.490607 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:18:15.490726 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 13:18:15.490802 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 13:18:15.490922 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260925 13:18:15.491590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 10678
I20260925 13:18:15.491683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:18:15.491677 10807 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40657
I20260925 13:18:15.491756 10807 heartbeater.cc:464] Registering TS with master...
I20260925 13:18:15.491840 392 external_mini_cluster.cc:1428] Setting key 77ce49295c0139c7160e57272c72ebdb
I20260925 13:18:15.491901 10807 heartbeater.cc:511] Master 127.0.98.62:40657 requested a full tablet report, sending...
I20260925 13:18:15.492414 10615 ts_manager.cc:194] Registered new tserver with Master: 1affb76bbb784605a874f035be495154 (127.0.98.1:35639)
I20260925 13:18:15.493229 10615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41805
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:18:15.514885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40657
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:18:15.594689 10812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:18:15.594920 10812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:18:15.594955 10812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:18:15.596180 10812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:18:15.596226 10812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:18:15.596263 10812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:18:15.597807 10812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:18:15.598068 10812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:18:15.598284 10812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:18:15.603005 10818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:15.603137 10820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:15.603175 10812 server_base.cc:1034] running on GCE node
W20260925 13:18:15.603121 10817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:15.603500 10812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:18:15.603710 10812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:18:15.604840 10812 hybrid_clock.cc:648] HybridClock initialized: now 1790342295604827 us; error 28 us; skew 500 ppm
Sep 25 13:18:15 dist-test-slave-t8d9 krb5kdc[10571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:18:15.607281 10812 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:18:15.607715 10812 webserver.cc:460] Webserver started at http://127.0.98.2:34139/ using document root <none> and password file <none>
I20260925 13:18:15.607906 10812 fs_manager.cc:359] Metadata directory not provided
I20260925 13:18:15.607945 10812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:18:15.608039 10812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:18:15.609031 10812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "a69e96154d074264bdd9a6c34a19eb46"
format_stamp: "Formatted at 2026-09-25 13:18:15 on dist-test-slave-t8d9"
server_key: "fc8a9a6fb8b58746aeb0547c4f0ca288"
server_key_iv: "bfa084dfc0f8c16e3292114f8321d731"
server_key_version: "encryptionkey@0"
I20260925 13:18:15.609328 10812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a69e96154d074264bdd9a6c34a19eb46"
format_stamp: "Formatted at 2026-09-25 13:18:15 on dist-test-slave-t8d9"
server_key: "fc8a9a6fb8b58746aeb0547c4f0ca288"
server_key_iv: "bfa084dfc0f8c16e3292114f8321d731"
server_key_version: "encryptionkey@0"
I20260925 13:18:15.610570 10812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:18:15.611171 10827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:15.611302 10812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:18:15.611379 10812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "a69e96154d074264bdd9a6c34a19eb46"
format_stamp: "Formatted at 2026-09-25 13:18:15 on dist-test-slave-t8d9"
server_key: "fc8a9a6fb8b58746aeb0547c4f0ca288"
server_key_iv: "bfa084dfc0f8c16e3292114f8321d731"
server_key_version: "encryptionkey@0"
I20260925 13:18:15.611447 10812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:18:15.638573 10812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:18:15.640787 10812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:18:15.640904 10812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:18:15.641137 10812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:18:15.641425 10812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:18:15.641465 10812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:15.641499 10812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:18:15.641521 10812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:15.646970 10812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44335
I20260925 13:18:15.647025 10940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44335 every 8 connection(s)
I20260925 13:18:15.647332 10812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:18:15.649880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 10812
I20260925 13:18:15.649977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:18:15.650163 392 external_mini_cluster.cc:1428] Setting key d6a0b045929fad6c849a7e56652688a2
Sep 25 13:18:15 dist-test-slave-t8d9 krb5kdc[10571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:18:15.656657 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:15.648051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48339 (local address 127.0.98.62:40657)
0925 13:18:15.648106 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:15.648107 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:15.648307 (+ 200us) server_negotiation.cc:407] Connection header received
0925 13:18:15.648805 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:15.648806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:15.648819 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:15.648870 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:15.649846 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.650440 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.651430 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.651629 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.651684 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:15.653886 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:18:15.653887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:18:15.653888 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:18:15.653897 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:18:15.655237 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:18:15.655690 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:18:15.655690 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:18:15.655691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:18:15.655740 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:18:15.656030 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:18:15.656031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:18:15.656031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:18:15.656151 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:18:15.656216 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:15.656526 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 13:18:15.656592 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:18:15.656924 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:15.648025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40657 (local address 127.0.98.2:48339)
0925 13:18:15.648189 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:15.648228 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:15.648711 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:15.648939 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:15.648943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:15.649516 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:15.649758 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:15.649763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.650531 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.650532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:15.651333 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:15.651335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.651699 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.651700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:15.651813 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:15.652293 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:15.652302 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:15.653789 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:18:15.655316 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:18:15.655322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:18:15.655325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:18:15.655617 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:18:15.655797 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:18:15.655797 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:18:15.655797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:18:15.655940 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:18:15.656223 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:18:15.656229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:18:15.656398 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 13:18:15.656514 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 13:18:15.656677 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260925 13:18:15.657362 10941 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40657
I20260925 13:18:15.657449 10941 heartbeater.cc:464] Registering TS with master...
I20260925 13:18:15.657663 10941 heartbeater.cc:511] Master 127.0.98.62:40657 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 13:18:15.658061 10615 ts_manager.cc:194] Registered new tserver with Master: a69e96154d074264bdd9a6c34a19eb46 (127.0.98.2:44335)
I20260925 13:18:15.658417 10615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:18:15.674008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40657
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:18:15.754580 10946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:18:15.754813 10946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:18:15.754839 10946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:18:15.756089 10946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:18:15.756141 10946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:18:15.756170 10946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:18:15.757745 10946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:18:15.757990 10946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:18:15.758250 10946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:18:15.763046 10954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:15.763041 10951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:15.763041 10952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:15.763410 10946 server_base.cc:1034] running on GCE node
I20260925 13:18:15.763594 10946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:18:15.763797 10946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:18:15.764930 10946 hybrid_clock.cc:648] HybridClock initialized: now 1790342295764915 us; error 28 us; skew 500 ppm
Sep 25 13:18:15 dist-test-slave-t8d9 krb5kdc[10571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:18:15.767546 10946 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:18:15.767977 10946 webserver.cc:460] Webserver started at http://127.0.98.3:39959/ using document root <none> and password file <none>
I20260925 13:18:15.768153 10946 fs_manager.cc:359] Metadata directory not provided
I20260925 13:18:15.768190 10946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:18:15.768281 10946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:18:15.769325 10946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "9d2e197506bb423d92ee13660bd66024"
format_stamp: "Formatted at 2026-09-25 13:18:15 on dist-test-slave-t8d9"
server_key: "293dd315440a509764c31968f9f223b3"
server_key_iv: "73e5e584428c5483ab33f03f4931c93d"
server_key_version: "encryptionkey@0"
I20260925 13:18:15.769680 10946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9d2e197506bb423d92ee13660bd66024"
format_stamp: "Formatted at 2026-09-25 13:18:15 on dist-test-slave-t8d9"
server_key: "293dd315440a509764c31968f9f223b3"
server_key_iv: "73e5e584428c5483ab33f03f4931c93d"
server_key_version: "encryptionkey@0"
I20260925 13:18:15.770905 10946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:18:15.771579 10961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:15.771732 10946 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:18:15.771813 10946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "9d2e197506bb423d92ee13660bd66024"
format_stamp: "Formatted at 2026-09-25 13:18:15 on dist-test-slave-t8d9"
server_key: "293dd315440a509764c31968f9f223b3"
server_key_iv: "73e5e584428c5483ab33f03f4931c93d"
server_key_version: "encryptionkey@0"
I20260925 13:18:15.771875 10946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:18:15.792476 10946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:18:15.794447 10946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:18:15.794605 10946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:18:15.794835 10946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:18:15.795126 10946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:18:15.795161 10946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:15.795188 10946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:18:15.795212 10946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:15.800803 10946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39871
I20260925 13:18:15.800843 11074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39871 every 8 connection(s)
I20260925 13:18:15.801143 10946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:18:15 dist-test-slave-t8d9 krb5kdc[10571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:18:15.809039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 10946
I20260925 13:18:15.809139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:18:15.809319 392 external_mini_cluster.cc:1428] Setting key 0317f93f6e207abd4ee93342d3d80999
I20260925 13:18:15.809559 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:15.801860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43777 (local address 127.0.98.62:40657)
0925 13:18:15.801915 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:15.801917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:15.802128 (+ 211us) server_negotiation.cc:407] Connection header received
0925 13:18:15.802593 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:15.802594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:15.802604 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:15.802652 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:15.803548 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.804037 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.804989 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.805162 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.805202 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:15.807239 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:18:15.807241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:18:15.807241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:18:15.807248 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:18:15.808335 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:18:15.808667 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:18:15.808668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:18:15.808668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:18:15.808712 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:18:15.808891 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:18:15.808892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:18:15.808892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:18:15.809037 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:18:15.809104 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:15.809387 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 13:18:15.809465 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:18:15.809768 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:15.801860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40657 (local address 127.0.98.3:43777)
0925 13:18:15.802009 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:15.802049 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:15.802515 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:15.802711 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:15.802714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:15.803279 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:15.803476 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:15.803482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.804103 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.804104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:15.804931 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:15.804933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.805214 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.805215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:15.805311 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:15.805838 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:15.805847 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:15.807151 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:18:15.808400 (+ 1249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:18:15.808406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:18:15.808407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:18:15.808612 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:18:15.808753 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:18:15.808754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:18:15.808754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:18:15.808843 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:18:15.809092 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:18:15.809095 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:18:15.809256 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 13:18:15.809341 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 13:18:15.809500 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":84,"threads_started":1}
I20260925 13:18:15.810209 11075 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40657
I20260925 13:18:15.810302 11075 heartbeater.cc:464] Registering TS with master...
I20260925 13:18:15.810473 11075 heartbeater.cc:511] Master 127.0.98.62:40657 requested a full tablet report, sending...
I20260925 13:18:15.810829 10615 ts_manager.cc:194] Registered new tserver with Master: 9d2e197506bb423d92ee13660bd66024 (127.0.98.3:39871)
I20260925 13:18:15.811195 10615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43777
I20260925 13:18:15.820986 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:18:15.827417 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:15.822258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54316 (local address 127.0.98.62:40657)
0925 13:18:15.822326 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:15.822329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:15.822399 (+ 70us) server_negotiation.cc:407] Connection header received
0925 13:18:15.822450 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:15.822451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:15.822466 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:15.822512 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:15.823022 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.823504 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.824196 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.824373 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.824413 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:15.825483 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:18:15.825484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:18:15.825484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:18:15.825488 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:18:15.826586 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:18:15.826852 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:18:15.826853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:18:15.826853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:18:15.826895 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:18:15.827013 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:18:15.827014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:18:15.827014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:18:15.827148 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:18:15.827213 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:15.827287 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 13:18:15.827345 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:18:15.828996 10615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:18:15.830554 10615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:18:15.836656 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:15.832755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44335 (local address 127.0.0.1:39166)
0925 13:18:15.833276 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:15.833283 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:15.833314 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:15.833619 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:15.833620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:15.833622 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:18:15.833663 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:15.833665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.834543 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.834544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:15.835493 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:15.835496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.836360 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.836361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:15.836437 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:15.836451 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 13:18:15.836521 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 13:18:15.836578 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":74,"thread_start_us":75,"threads_started":1}
I20260925 13:18:15.836671 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:15.832990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.0.98.2:44335)
0925 13:18:15.833365 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:15.833370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:15.833378 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:18:15.833411 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:15.833413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:15.833427 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:15.833520 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:18:15.833736 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.834443 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.835633 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.836287 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.836353 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:15.836416 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:15.836547 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 13:18:15.836587 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":76,"threads_started":1}
I20260925 13:18:15.837213 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:15.832755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35639 (local address 127.0.0.1:34870)
0925 13:18:15.833161 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:15.833170 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:15.833215 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:15.833391 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:15.833392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:15.833393 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:18:15.833448 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:15.833451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.834721 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.834722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:15.836007 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:15.836009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.836978 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.836979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:15.837052 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:15.837055 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:18:15.837099 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 13:18:15.837140 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":50,"thread_start_us":85,"threads_started":1}
I20260925 13:18:15.837363 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:15.832835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.0.98.3:39871)
0925 13:18:15.833015 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:15.833020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:15.833034 (+ 14us) server_negotiation.cc:407] Connection header received
0925 13:18:15.833142 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:15.833145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:15.833163 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:15.833240 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:18:15.833440 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.834619 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.836084 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.836812 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.836857 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:15.836911 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:15.837240 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 13:18:15.837287 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
I20260925 13:18:15.837294 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:15.832752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39871 (local address 127.0.0.1:39592)
0925 13:18:15.832938 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:15.832991 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:15.833055 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:15.833266 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:15.833270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:15.833275 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:18:15.833350 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:15.833352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.834721 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.834722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:15.835979 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:15.835982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.837072 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.837073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:15.837140 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:15.837143 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:18:15.837211 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 13:18:15.837243 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":58,"thread_start_us":50,"threads_started":1}
I20260925 13:18:15.837721 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:15.833005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34870 (local address 127.0.98.1:35639)
0925 13:18:15.833175 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:15.833178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:15.833208 (+ 30us) server_negotiation.cc:407] Connection header received
0925 13:18:15.833292 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:15.833295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:15.833310 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:15.833394 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:18:15.833503 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.834606 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.836084 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:15.836812 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:15.836875 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:15.836946 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:15.837599 (+ 653us) server_negotiation.cc:299] Negotiation successful
0925 13:18:15.837645 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
I20260925 13:18:15.837673 10875 tablet_service.cc:1467] Processing CreateTablet for tablet eafce48e235c4ece9a6b60dd6a6aaeaf (DEFAULT_TABLE table=test-table [id=99120f00930d41c6a1b000d3037e96d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:18:15.838047 10875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eafce48e235c4ece9a6b60dd6a6aaeaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:18:15.838210 11009 tablet_service.cc:1467] Processing CreateTablet for tablet eafce48e235c4ece9a6b60dd6a6aaeaf (DEFAULT_TABLE table=test-table [id=99120f00930d41c6a1b000d3037e96d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:18:15.838482 11009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eafce48e235c4ece9a6b60dd6a6aaeaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:18:15.838397 10741 tablet_service.cc:1467] Processing CreateTablet for tablet eafce48e235c4ece9a6b60dd6a6aaeaf (DEFAULT_TABLE table=test-table [id=99120f00930d41c6a1b000d3037e96d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:18:15.838689 10741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eafce48e235c4ece9a6b60dd6a6aaeaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:18:15.840914 11092 tablet_bootstrap.cc:492] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024: Bootstrap starting.
I20260925 13:18:15.841621 11092 tablet_bootstrap.cc:654] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024: Neither blocks nor log segments found. Creating new log.
I20260925 13:18:15.841790 11093 tablet_bootstrap.cc:492] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46: Bootstrap starting.
I20260925 13:18:15.842198 11092 log.cc:824] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024: Log is configured to *not* fsync() on all Append() calls
I20260925 13:18:15.842419 11093 tablet_bootstrap.cc:654] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46: Neither blocks nor log segments found. Creating new log.
I20260925 13:18:15.842922 11093 log.cc:824] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46: Log is configured to *not* fsync() on all Append() calls
I20260925 13:18:15.842960 11092 tablet_bootstrap.cc:492] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024: No bootstrap required, opened a new log
I20260925 13:18:15.843025 11092 ts_tablet_manager.cc:1397] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 13:18:15.843250 11094 tablet_bootstrap.cc:492] T eafce48e235c4ece9a6b60dd6a6aaeaf P 1affb76bbb784605a874f035be495154: Bootstrap starting.
I20260925 13:18:15.843568 11093 tablet_bootstrap.cc:492] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46: No bootstrap required, opened a new log
I20260925 13:18:15.843628 11093 ts_tablet_manager.cc:1397] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:18:15.843900 11094 tablet_bootstrap.cc:654] T eafce48e235c4ece9a6b60dd6a6aaeaf P 1affb76bbb784605a874f035be495154: Neither blocks nor log segments found. Creating new log.
I20260925 13:18:15.844414 11094 log.cc:824] T eafce48e235c4ece9a6b60dd6a6aaeaf P 1affb76bbb784605a874f035be495154: Log is configured to *not* fsync() on all Append() calls
I20260925 13:18:15.844903 11092 raft_consensus.cc:348] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2e197506bb423d92ee13660bd66024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "1affb76bbb784605a874f035be495154" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "a69e96154d074264bdd9a6c34a19eb46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44335 } }
I20260925 13:18:15.845050 11092 raft_consensus.cc:374] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:18:15.845073 11092 raft_consensus.cc:729] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2e197506bb423d92ee13660bd66024, State: Initialized, Role: FOLLOWER
I20260925 13:18:15.845175 11092 consensus_queue.cc:260] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2e197506bb423d92ee13660bd66024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "1affb76bbb784605a874f035be495154" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "a69e96154d074264bdd9a6c34a19eb46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44335 } }
I20260925 13:18:15.845224 11093 raft_consensus.cc:348] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2e197506bb423d92ee13660bd66024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "1affb76bbb784605a874f035be495154" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "a69e96154d074264bdd9a6c34a19eb46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44335 } }
I20260925 13:18:15.845383 11093 raft_consensus.cc:374] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:18:15.845438 11092 ts_tablet_manager.cc:1428] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 13:18:15.845440 11093 raft_consensus.cc:729] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69e96154d074264bdd9a6c34a19eb46, State: Initialized, Role: FOLLOWER
I20260925 13:18:15.845458 11075 heartbeater.cc:503] Master 127.0.98.62:40657 was elected leader, sending a full tablet report...
I20260925 13:18:15.845587 11093 consensus_queue.cc:260] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2e197506bb423d92ee13660bd66024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "1affb76bbb784605a874f035be495154" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "a69e96154d074264bdd9a6c34a19eb46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44335 } }
I20260925 13:18:15.845911 11093 ts_tablet_manager.cc:1428] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 13:18:15.846024 10941 heartbeater.cc:503] Master 127.0.98.62:40657 was elected leader, sending a full tablet report...
I20260925 13:18:15.846359 11094 tablet_bootstrap.cc:492] T eafce48e235c4ece9a6b60dd6a6aaeaf P 1affb76bbb784605a874f035be495154: No bootstrap required, opened a new log
I20260925 13:18:15.846422 11094 ts_tablet_manager.cc:1397] T eafce48e235c4ece9a6b60dd6a6aaeaf P 1affb76bbb784605a874f035be495154: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 13:18:15.847780 11094 raft_consensus.cc:348] T eafce48e235c4ece9a6b60dd6a6aaeaf P 1affb76bbb784605a874f035be495154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2e197506bb423d92ee13660bd66024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "1affb76bbb784605a874f035be495154" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "a69e96154d074264bdd9a6c34a19eb46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44335 } }
I20260925 13:18:15.847916 11094 raft_consensus.cc:374] T eafce48e235c4ece9a6b60dd6a6aaeaf P 1affb76bbb784605a874f035be495154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:18:15.847937 11094 raft_consensus.cc:729] T eafce48e235c4ece9a6b60dd6a6aaeaf P 1affb76bbb784605a874f035be495154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1affb76bbb784605a874f035be495154, State: Initialized, Role: FOLLOWER
I20260925 13:18:15.848019 11094 consensus_queue.cc:260] T eafce48e235c4ece9a6b60dd6a6aaeaf P 1affb76bbb784605a874f035be495154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2e197506bb423d92ee13660bd66024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "1affb76bbb784605a874f035be495154" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "a69e96154d074264bdd9a6c34a19eb46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44335 } }
I20260925 13:18:15.848213 11094 ts_tablet_manager.cc:1428] T eafce48e235c4ece9a6b60dd6a6aaeaf P 1affb76bbb784605a874f035be495154: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:18:15.848270 10807 heartbeater.cc:503] Master 127.0.98.62:40657 was elected leader, sending a full tablet report...
W20260925 13:18:15.897913 10942 tablet.cc:2369] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:18:15.983192 10808 tablet.cc:2369] T eafce48e235c4ece9a6b60dd6a6aaeaf P 1affb76bbb784605a874f035be495154: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:18:16.051705 11076 tablet.cc:2369] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:18:16.082146 11098 raft_consensus.cc:482] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:18:16.082273 11098 raft_consensus.cc:504] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2e197506bb423d92ee13660bd66024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "1affb76bbb784605a874f035be495154" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "a69e96154d074264bdd9a6c34a19eb46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44335 } }
I20260925 13:18:16.082588 11098 leader_election.cc:288] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1affb76bbb784605a874f035be495154 (127.0.98.1:35639), a69e96154d074264bdd9a6c34a19eb46 (127.0.98.2:44335)
I20260925 13:18:16.085232 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:16.082776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35639 (local address 127.0.98.3:46417)
0925 13:18:16.082843 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:16.082856 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:16.082922 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:16.083049 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:16.083049 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:16.083051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:18:16.083117 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:16.083118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:16.083748 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:16.083748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:16.084492 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:16.084494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:16.084996 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:16.084996 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:16.085072 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:16.085075 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:18:16.085119 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 13:18:16.085159 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 13:18:16.085232 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:16.082891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46417 (local address 127.0.98.1:35639)
0925 13:18:16.082934 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:16.082935 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:16.082946 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:18:16.082975 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:16.082976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:16.082987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:16.083040 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:18:16.083178 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:16.083670 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:16.084581 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:16.084929 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:16.084987 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:16.084996 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:16.085134 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 13:18:16.085167 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:18:16.085333 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:16.082903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34979 (local address 127.0.98.2:44335)
0925 13:18:16.082949 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:16.082952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:16.083030 (+ 78us) server_negotiation.cc:407] Connection header received
0925 13:18:16.083092 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:16.083093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:16.083104 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:16.083151 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:18:16.083298 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:16.083757 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:16.084666 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:16.085028 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:16.085087 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:16.085098 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:16.085233 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 13:18:16.085270 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:18:16.085350 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:16.082783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44335 (local address 127.0.98.3:34979)
0925 13:18:16.082972 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:16.082981 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:16.083039 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:16.083167 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:16.083168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:16.083169 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:18:16.083239 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:16.083240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:16.083833 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:16.083834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:16.084581 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:16.084583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:16.085100 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:16.085101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:16.085172 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:16.085175 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:18:16.085218 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:18:16.085256 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":16,"thread_start_us":100,"threads_started":1}
I20260925 13:18:16.085596 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafce48e235c4ece9a6b60dd6a6aaeaf" candidate_uuid: "9d2e197506bb423d92ee13660bd66024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1affb76bbb784605a874f035be495154" is_pre_election: true
I20260925 13:18:16.085639 10895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafce48e235c4ece9a6b60dd6a6aaeaf" candidate_uuid: "9d2e197506bb423d92ee13660bd66024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69e96154d074264bdd9a6c34a19eb46" is_pre_election: true
I20260925 13:18:16.085758 10761 raft_consensus.cc:2454] T eafce48e235c4ece9a6b60dd6a6aaeaf P 1affb76bbb784605a874f035be495154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d2e197506bb423d92ee13660bd66024 in term 0.
I20260925 13:18:16.085781 10895 raft_consensus.cc:2454] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d2e197506bb423d92ee13660bd66024 in term 0.
I20260925 13:18:16.085988 10962 leader_election.cc:302] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d2e197506bb423d92ee13660bd66024, a69e96154d074264bdd9a6c34a19eb46; no voters:
I20260925 13:18:16.086119 11098 raft_consensus.cc:2781] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:18:16.086172 11098 raft_consensus.cc:482] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:18:16.086195 11098 raft_consensus.cc:3037] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:18:16.086856 11098 raft_consensus.cc:504] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2e197506bb423d92ee13660bd66024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "1affb76bbb784605a874f035be495154" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "a69e96154d074264bdd9a6c34a19eb46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44335 } }
I20260925 13:18:16.086968 11098 leader_election.cc:288] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [CANDIDATE]: Term 1 election: Requested vote from peers 1affb76bbb784605a874f035be495154 (127.0.98.1:35639), a69e96154d074264bdd9a6c34a19eb46 (127.0.98.2:44335)
I20260925 13:18:16.087143 10761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafce48e235c4ece9a6b60dd6a6aaeaf" candidate_uuid: "9d2e197506bb423d92ee13660bd66024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1affb76bbb784605a874f035be495154"
I20260925 13:18:16.087154 10895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafce48e235c4ece9a6b60dd6a6aaeaf" candidate_uuid: "9d2e197506bb423d92ee13660bd66024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69e96154d074264bdd9a6c34a19eb46"
I20260925 13:18:16.087219 10895 raft_consensus.cc:3037] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:18:16.087218 10761 raft_consensus.cc:3037] T eafce48e235c4ece9a6b60dd6a6aaeaf P 1affb76bbb784605a874f035be495154 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:18:16.087916 10895 raft_consensus.cc:2454] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d2e197506bb423d92ee13660bd66024 in term 1.
I20260925 13:18:16.087916 10761 raft_consensus.cc:2454] T eafce48e235c4ece9a6b60dd6a6aaeaf P 1affb76bbb784605a874f035be495154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d2e197506bb423d92ee13660bd66024 in term 1.
I20260925 13:18:16.088073 10962 leader_election.cc:302] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d2e197506bb423d92ee13660bd66024, a69e96154d074264bdd9a6c34a19eb46; no voters:
I20260925 13:18:16.088165 11098 raft_consensus.cc:2781] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:18:16.088321 11098 raft_consensus.cc:686] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [term 1 LEADER]: Becoming Leader. State: Replica: 9d2e197506bb423d92ee13660bd66024, State: Running, Role: LEADER
I20260925 13:18:16.088410 11098 consensus_queue.cc:237] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2e197506bb423d92ee13660bd66024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "1affb76bbb784605a874f035be495154" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } } peers { permanent_uuid: "a69e96154d074264bdd9a6c34a19eb46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44335 } }
I20260925 13:18:16.089232 10616 catalog_manager.cc:5495] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d2e197506bb423d92ee13660bd66024 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9d2e197506bb423d92ee13660bd66024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2e197506bb423d92ee13660bd66024" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1affb76bbb784605a874f035be495154" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69e96154d074264bdd9a6c34a19eb46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44335 } health_report { overall_health: UNKNOWN } } }
I20260925 13:18:16.121887 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:16.119524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39596 (local address 127.0.98.3:39871)
0925 13:18:16.119591 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:16.119592 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:16.119600 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:18:16.119637 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:16.119638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:16.119659 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:16.119720 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:18:16.119851 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:16.120336 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:16.121087 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:16.121275 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:16.121331 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:16.121484 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:18:16.121660 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:18:16.121743 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:16.121766 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 13:18:16.121789 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 13:18:16.123795 10895 raft_consensus.cc:1261] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46 [term 1 FOLLOWER]: Refusing update from remote peer 9d2e197506bb423d92ee13660bd66024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:18:16.123795 10761 raft_consensus.cc:1261] T eafce48e235c4ece9a6b60dd6a6aaeaf P 1affb76bbb784605a874f035be495154 [term 1 FOLLOWER]: Refusing update from remote peer 9d2e197506bb423d92ee13660bd66024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:18:16.124140 11098 consensus_queue.cc:1035] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1affb76bbb784605a874f035be495154" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:18:16.124302 11102 consensus_queue.cc:1035] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69e96154d074264bdd9a6c34a19eb46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:18:16.126478 11105 mvcc.cc:205] Tried to move back new op lower bound from 7333242044920438784 to 7333242044778692608. Current Snapshot: MvccSnapshot[applied={T|T < 7333242044920438784}]
I20260925 13:18:16.126525 11106 mvcc.cc:205] Tried to move back new op lower bound from 7333242044920438784 to 7333242044778692608. Current Snapshot: MvccSnapshot[applied={T|T < 7333242044920438784}]
I20260925 13:18:16.128898 10616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:18:16.129004 10616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:18:16.129751 10616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe3f3fa572064a9188e281f6d803288b: Sending DeleteTablet for 3 replicas of tablet eafce48e235c4ece9a6b60dd6a6aaeaf
I20260925 13:18:16.130069 10875 tablet_service.cc:1514] Processing DeleteTablet for tablet eafce48e235c4ece9a6b60dd6a6aaeaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:18:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39166
I20260925 13:18:16.130067 11009 tablet_service.cc:1514] Processing DeleteTablet for tablet eafce48e235c4ece9a6b60dd6a6aaeaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:18:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39592
I20260925 13:18:16.130188 10741 tablet_service.cc:1514] Processing DeleteTablet for tablet eafce48e235c4ece9a6b60dd6a6aaeaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:18:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34870
I20260925 13:18:16.130306 11115 tablet_replica.cc:330] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46: stopping tablet replica
I20260925 13:18:16.130410 11115 raft_consensus.cc:2229] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:18:16.130503 11115 raft_consensus.cc:2258] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:18:16.130682 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 10678
I20260925 13:18:16.131803 11116 tablet_replica.cc:330] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024: stopping tablet replica
I20260925 13:18:16.131913 11116 raft_consensus.cc:2229] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:18:16.132031 11116 raft_consensus.cc:2258] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:18:16.132339 11115 ts_tablet_manager.cc:1905] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:18:16.134333 11116 ts_tablet_manager.cc:1905] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:18:16.135816 11116 ts_tablet_manager.cc:1918] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:18:16.135897 11116 log.cc:1196] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/eafce48e235c4ece9a6b60dd6a6aaeaf
I20260925 13:18:16.136132 11116 ts_tablet_manager.cc:1939] T eafce48e235c4ece9a6b60dd6a6aaeaf P 9d2e197506bb423d92ee13660bd66024: Deleting consensus metadata
I20260925 13:18:16.136540 11115 ts_tablet_manager.cc:1918] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:18:16.136641 11115 log.cc:1196] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/eafce48e235c4ece9a6b60dd6a6aaeaf
I20260925 13:18:16.136663 10601 catalog_manager.cc:4841] TS 9d2e197506bb423d92ee13660bd66024 (127.0.98.3:39871): tablet eafce48e235c4ece9a6b60dd6a6aaeaf (table test-table [id=99120f00930d41c6a1b000d3037e96d6]) successfully deleted
I20260925 13:18:16.136904 11115 ts_tablet_manager.cc:1939] T eafce48e235c4ece9a6b60dd6a6aaeaf P a69e96154d074264bdd9a6c34a19eb46: Deleting consensus metadata
I20260925 13:18:16.137066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 10812
W20260925 13:18:16.136735 10603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35639 (error 108)
I20260925 13:18:16.138064 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:16.137945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35639 (local address 127.0.0.1:34872)
0925 13:18:16.137996 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:16.138030 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 13:18:16.138175 10603 catalog_manager.cc:4584] TS 1affb76bbb784605a874f035be495154 (127.0.98.1:35639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eafce48e235c4ece9a6b60dd6a6aaeaf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35639: connect: Connection refused (error 111)
I20260925 13:18:16.141434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 10946
I20260925 13:18:16.145888 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 10584
I20260925 13:18:16.146045 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:16.145891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44335 (local address 127.0.0.1:39172)
0925 13:18:16.145961 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:16.145994 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
2026-09-25T13:18:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2065 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:18:16 dist-test-slave-t8d9 krb5kdc[11121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:18:16 dist-test-slave-t8d9 krb5kdc[11121](info): set up 2 sockets
Sep 25 13:18:16 dist-test-slave-t8d9 krb5kdc[11121](info): commencing operation
krb5kdc: starting...
W20260925 13:18:17.094112 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:18:17 dist-test-slave-t8d9 krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:18:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:18:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 13:18:17.245951 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskC7bpD2/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/postgres -l logfile start
2026-09-25 13:18:19.592 UTC [11151] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 13:18:19.592 UTC [11151] LOG: listening on IPv4 address "127.0.98.20", port 51181
2026-09-25 13:18:19.595 UTC [11151] LOG: listening on Unix socket "/tmp/.s.PGSQL.51181"
2026-09-25 13:18:19.614 UTC [11154] LOG: database system was shut down at 2026-09-25 13:18:18 UTC
2026-09-25 13:18:19.618 UTC [11151] LOG: database system is ready to accept connections
I20260925 13:18:20.490012 392 mini_postgres.cc:96] Postgres bound to 51181
2026-09-25 13:18:20.495 UTC [11162] FATAL: database "slave" does not exist
127.0.98.20:51181 - accepting connections
I20260925 13:18:20.496222 392 mini_ranger.cc:161] Starting Ranger...
I20260925 13:18:20.510358 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 13:18:20.956012 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 13:18:20.956096 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-admin
2026-09-25 13:18:21,227 [I] DB FLAVOR :POSTGRES
2026-09-25 13:18:21,227 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 13:18:21,227 [I] Checking connection..
2026-09-25 13:18:21,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 13:18:21,484 [I] Checking connection passed.
2026-09-25 13:18:21,485 [I] --------- Verifying version history table ---------
2026-09-25 13:18:21,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 13:18:21,715 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 13:18:21,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 13:18:21,949 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 13:18:21,949 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 13:18:21,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 13:18:22.188 UTC [11226] WARNING: there is no transaction in progress
2026-09-25 13:18:22,193 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 13:18:22,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 13:18:22,445 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 13:18:22,445 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 13:18:22,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 13:18:22,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 13:18:22,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-t8d9.c.gcp-upstream.internal','N') ;"
2026-09-25 13:18:23,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 13:18:23,400 [I] Table x_portal_user does not exist in database ranger
2026-09-25 13:18:23,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 13:18:23,646 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 13:18:23,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 13:18:23,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 13:18:24,154 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 13:18:24,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 13:18:25.102 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25.118 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25.135 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25.150 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25.167 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25.184 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25.240 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25.246 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25.261 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25.271 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25.282 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25.296 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25.306 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25.313 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25.320 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25.326 UTC [11334] WARNING: there is no transaction in progress
2026-09-25 13:18:25,968 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 13:18:25,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 13:18:26,209 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 13:18:26,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 13:18:26,465 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 13:18:26,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 13:18:26,722 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 13:18:26,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 13:18:26,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 13:18:27,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-t8d9.c.gcp-upstream.internal';"
2026-09-25 13:18:27,442 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 13:18:27,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 13:18:27,687 [I] DB_PATCHES have already been applied
I20260925 13:18:27.693046 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskC7bpD2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 13:18:27.693110 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 13:18:27.695915 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:59531
Sep 25, 2026 1:18:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:18:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:18:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 1:18:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 1:18:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 1:18:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 1:18:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 13:18:28.679932 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.184s user 0.000s sys 0.008s
Sep 25, 2026 1:18:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 1:18:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 1:18:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 1:18:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 1:18:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 1:18:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 1:18:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 1:18:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 1:18:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 1:18:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 1:18:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 1:18:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 1:18:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 1:18:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 1:18:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 1:18:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 1:18:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 1:18:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 13:18:50.275701 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 13:18:50 dist-test-slave-t8d9 krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342330, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 13:18:50.356770 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 13:18:50.365935 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 13:18:51.135515 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 13:18:51.135598 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-kms
I20260925 13:18:51.135617 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:51181
2026-09-25 13:18:51,348 [I] DB FLAVOR :POSTGRES
2026-09-25 13:18:51,348 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 13:18:51,348 [I] Checking connection
2026-09-25 13:18:51,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 13:18:51,617 [I] connection success
2026-09-25 13:18:51,618 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 13:18:51,618 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 13:18:51,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 13:18:51,893 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 13:18:51,893 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 13:18:51,893 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 13:18:51,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51181/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 13:18:52,176 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 13:18:52.612039 392 mini_ranger_kms.cc:310] Created kms service
I20260925 13:18:52.788161 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 13:18:52.830428 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 13:18:52 dist-test-slave-t8d9 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342330, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:18:53.213837 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 13:18:53.213950 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 13:18:53.213971 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 13:18:53.217434 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11514
I20260925 13:18:53.217571 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:58545
Sep 25, 2026 1:18:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:18:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:18:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 1:18:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 1:18:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 1:18:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 13:18:55.277593 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.921s user 0.006s sys 0.188s
I20260925 13:18:55.277698 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 13:18:55.277760 392 mini_ranger_kms.cc:401] 127.0.98.20:58545/kms/v1/keys
Sep 25 13:18:55 dist-test-slave-t8d9 krb5kdc[11121](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790342335, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 13:18:55 dist-test-slave-t8d9 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342335, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 13:18:56 dist-test-slave-t8d9 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342335, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 13:18:58 dist-test-slave-t8d9 krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:18:58.651345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34609
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34609
--ranger_config_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58545/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:18:58.731503 11565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:18:58.731707 11565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:18:58.731727 11565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:18:58.732836 11565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:18:58.732872 11565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:18:58.732884 11565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:18:58.732894 11565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:18:58.732904 11565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:18:58.734449 11565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:58545/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34609
--ranger_config_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:18:58.734633 11565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:18:58.734798 11565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:18:58.739511 11565 server_base.cc:1034] running on GCE node
W20260925 13:18:58.739535 11570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:58.739629 11573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:58.739543 11571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:58.739897 11565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:18:58.740126 11565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:18:58.741259 11565 hybrid_clock.cc:648] HybridClock initialized: now 1790342338741240 us; error 28 us; skew 500 ppm
Sep 25 13:18:58 dist-test-slave-t8d9 krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:18:58.748260 11565 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:18:58.748653 11565 webserver.cc:460] Webserver started at http://127.0.98.62:41975/ using document root <none> and password file <none>
I20260925 13:18:58.748840 11565 fs_manager.cc:359] Metadata directory not provided
I20260925 13:18:58.748876 11565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:18:58.748961 11565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:18:58 dist-test-slave-t8d9 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:18:58.809789 11565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "9aa1222235264e8c8bdef7fd2fa9f373"
format_stamp: "Formatted at 2026-09-25 13:18:58 on dist-test-slave-t8d9"
server_key: "f7cf779eb71ee0556565f9c38a3d8cdb"
server_key_iv: "9cb1032ca1ef1ca0c495e94e14c0eacf"
server_key_version: "kuduclusterkey@0"
I20260925 13:18:58.810220 11565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "9aa1222235264e8c8bdef7fd2fa9f373"
format_stamp: "Formatted at 2026-09-25 13:18:58 on dist-test-slave-t8d9"
server_key: "f7cf779eb71ee0556565f9c38a3d8cdb"
server_key_iv: "9cb1032ca1ef1ca0c495e94e14c0eacf"
server_key_version: "kuduclusterkey@0"
I20260925 13:18:58.811667 11565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:18:58.827294 11580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:58.827546 11565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:18:58.827652 11565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "9aa1222235264e8c8bdef7fd2fa9f373"
format_stamp: "Formatted at 2026-09-25 13:18:58 on dist-test-slave-t8d9"
server_key: "f7cf779eb71ee0556565f9c38a3d8cdb"
server_key_iv: "9cb1032ca1ef1ca0c495e94e14c0eacf"
server_key_version: "kuduclusterkey@0"
I20260925 13:18:58.827762 11565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:18:58.839928 11565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:18:58.841792 11565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:18:58.841910 11565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:18:58.846674 11565 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34609
I20260925 13:18:58.846720 11632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34609 every 8 connection(s)
I20260925 13:18:58.847133 11565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:18:58.847877 11633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:18:58.850764 11633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373: Bootstrap starting.
I20260925 13:18:58.851342 11633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373: Neither blocks nor log segments found. Creating new log.
I20260925 13:18:58.851583 11633 log.cc:824] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373: Log is configured to *not* fsync() on all Append() calls
I20260925 13:18:58.852365 11633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373: No bootstrap required, opened a new log
I20260925 13:18:58.854914 11633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa1222235264e8c8bdef7fd2fa9f373" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } }
I20260925 13:18:58.855077 11633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:18:58.855110 11633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa1222235264e8c8bdef7fd2fa9f373, State: Initialized, Role: FOLLOWER
I20260925 13:18:58.855254 11633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa1222235264e8c8bdef7fd2fa9f373" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } }
I20260925 13:18:58.855333 11633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:18:58.855365 11633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:18:58.855412 11633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:18:58.855993 11633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa1222235264e8c8bdef7fd2fa9f373" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } }
I20260925 13:18:58.856092 11633 leader_election.cc:302] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9aa1222235264e8c8bdef7fd2fa9f373; no voters:
I20260925 13:18:58.856274 11633 leader_election.cc:288] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:18:58.856305 11636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:18:58.856412 11636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [term 1 LEADER]: Becoming Leader. State: Replica: 9aa1222235264e8c8bdef7fd2fa9f373, State: Running, Role: LEADER
I20260925 13:18:58.856554 11633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:18:58.856561 11636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa1222235264e8c8bdef7fd2fa9f373" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } }
I20260925 13:18:58.856979 11637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9aa1222235264e8c8bdef7fd2fa9f373" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa1222235264e8c8bdef7fd2fa9f373" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } }
I20260925 13:18:58.857019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 11565
I20260925 13:18:58.857118 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:18:58.857231 11637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [sys.catalog]: This master's current role is: LEADER
I20260925 13:18:58.857307 392 external_mini_cluster.cc:1428] Setting key dde55db49d34ca7f4f4fd3e9a017a6f1
I20260925 13:18:58.857164 11638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9aa1222235264e8c8bdef7fd2fa9f373. Latest consensus state: current_term: 1 leader_uuid: "9aa1222235264e8c8bdef7fd2fa9f373" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa1222235264e8c8bdef7fd2fa9f373" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34609 } } }
I20260925 13:18:58.857370 11638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373 [sys.catalog]: This master's current role is: LEADER
I20260925 13:18:58.858007 11633 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 13:18:58.860338 11645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:18:58.861065 11645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:18:58.863301 11645 catalog_manager.cc:1347] Generated new cluster ID: 1b8970ad21cd430590be98a69c96a467
I20260925 13:18:58.863348 11645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:18:58.875111 11645 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:18:58.875912 11645 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:18:58.881896 11645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373: Generated new TSK 0
I20260925 13:18:58.882159 11645 catalog_manager.cc:1514] Initializing in-progress tserver states...
Sep 25 13:18:58 dist-test-slave-t8d9 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:18:58.889456 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:58.857861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.0.98.62:34609)
0925 13:18:58.865909 (+ 8048us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:58.865921 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:58.865959 (+ 38us) server_negotiation.cc:407] Connection header received
0925 13:18:58.866089 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:58.866097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:58.866160 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:58.866326 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:58.867367 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:58.870153 (+ 2786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:58.881294 (+ 11141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:58.882007 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:58.882148 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:58.885021 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:18:58.885027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:18:58.885027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:18:58.885041 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:18:58.887697 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:18:58.888115 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:18:58.888120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:18:58.888120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:18:58.888246 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:18:58.888438 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:18:58.888439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:18:58.888439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:18:58.888766 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:18:58.888908 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:58.889054 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 13:18:58.889207 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7978,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:18:59.641918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34609
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:18:59.744877 11678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:18:59.745097 11678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:18:59.745126 11678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:18:59.746847 11678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:18:59.746907 11678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:18:59.746953 11678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:18:59.749033 11678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:58545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:18:59.749279 11678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:18:59.749485 11678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:18:59.755235 11684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:59.755354 11683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:59.755268 11686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:59.755568 11678 server_base.cc:1034] running on GCE node
I20260925 13:18:59.755790 11678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:18:59.756031 11678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:18:59.759341 11678 hybrid_clock.cc:648] HybridClock initialized: now 1790342339759313 us; error 36 us; skew 500 ppm
Sep 25 13:18:59 dist-test-slave-t8d9 krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:18:59.761997 11678 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:18:59.762601 11678 webserver.cc:460] Webserver started at http://127.0.98.1:34141/ using document root <none> and password file <none>
I20260925 13:18:59.762852 11678 fs_manager.cc:359] Metadata directory not provided
I20260925 13:18:59.762902 11678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:18:59.763057 11678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:18:59 dist-test-slave-t8d9 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:18:59.778016 11678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "035fc3b36aac45dca47a47acbd0363f7"
format_stamp: "Formatted at 2026-09-25 13:18:59 on dist-test-slave-t8d9"
server_key: "42b28ddaf7d4a8f40e0d390add63d3e4"
server_key_iv: "fb6e07bb742213f478ff1b55ac4aed47"
server_key_version: "kuduclusterkey@0"
I20260925 13:18:59.778491 11678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "035fc3b36aac45dca47a47acbd0363f7"
format_stamp: "Formatted at 2026-09-25 13:18:59 on dist-test-slave-t8d9"
server_key: "42b28ddaf7d4a8f40e0d390add63d3e4"
server_key_iv: "fb6e07bb742213f478ff1b55ac4aed47"
server_key_version: "kuduclusterkey@0"
I20260925 13:18:59.780259 11678 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 13:18:59.794947 11695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:59.795187 11678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 13:18:59.795327 11678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "035fc3b36aac45dca47a47acbd0363f7"
format_stamp: "Formatted at 2026-09-25 13:18:59 on dist-test-slave-t8d9"
server_key: "42b28ddaf7d4a8f40e0d390add63d3e4"
server_key_iv: "fb6e07bb742213f478ff1b55ac4aed47"
server_key_version: "kuduclusterkey@0"
I20260925 13:18:59.795421 11678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:18:59.809633 11678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:18:59.812021 11678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:18:59.812179 11678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:18:59.812434 11678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:18:59.812824 11678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:18:59.812927 11678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:59.813022 11678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:18:59.813091 11678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:18:59.820108 11678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37195
I20260925 13:18:59.820544 11678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:18:59.827400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 11678
I20260925 13:18:59.827502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:18:59.827698 392 external_mini_cluster.cc:1428] Setting key 6898a7f0ddfe82de24271320f749f9ce
Sep 25 13:18:59 dist-test-slave-t8d9 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:18:59.829730 11808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37195 every 8 connection(s)
I20260925 13:18:59.832237 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:59.821279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34609 (local address 127.0.98.1:34709)
0925 13:18:59.821444 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 13:18:59.821491 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:18:59.822122 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:18:59.822540 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:18:59.822545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:18:59.822776 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:18:59.823038 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:59.823044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:59.824060 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:59.824061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:59.825366 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:18:59.825369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:18:59.825915 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:18:59.825917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:18:59.826061 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:59.826736 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:18:59.826747 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:18:59.828390 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:18:59.830629 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:18:59.830633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:18:59.830636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:18:59.830928 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:18:59.831142 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:18:59.831143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:18:59.831143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:18:59.831271 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:18:59.831532 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:18:59.831538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:18:59.831730 (+ 192us) client_negotiation.cc:769] Sending connection context
0925 13:18:59.831822 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 13:18:59.831988 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260925 13:18:59.832895 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:18:59.821591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34709 (local address 127.0.98.62:34609)
0925 13:18:59.822306 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0925 13:18:59.822308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:18:59.822316 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:18:59.822342 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:18:59.822342 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:18:59.822355 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:18:59.822421 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:18:59.823143 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:59.823962 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:59.825453 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:18:59.825815 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:18:59.826514 (+ 699us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:18:59.828488 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:18:59.828490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:18:59.828490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:18:59.828498 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:18:59.830486 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:18:59.831015 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:18:59.831016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:18:59.831016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:18:59.831080 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:18:59.831335 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:18:59.831335 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:18:59.831336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:18:59.831468 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:18:59.832649 (+ 1181us) server_negotiation.cc:1035] Waiting for connection context
0925 13:18:59.832707 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 13:18:59.832798 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":71,"threads_started":1}
I20260925 13:18:59.833551 11809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34609
I20260925 13:18:59.833647 11809 heartbeater.cc:464] Registering TS with master...
I20260925 13:18:59.833860 11809 heartbeater.cc:511] Master 127.0.98.62:34609 requested a full tablet report, sending...
I20260925 13:18:59.834504 11597 ts_manager.cc:194] Registered new tserver with Master: 035fc3b36aac45dca47a47acbd0363f7 (127.0.98.1:37195)
I20260925 13:18:59.835397 11597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34709
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:18:59.860671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34609
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 13:18:59 dist-test-slave-t8d9 krb5kdc[11121](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790342339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 13:18:59.976815 11815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:18:59.977048 11815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:18:59.977082 11815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:18:59.978444 11815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:18:59.978554 11815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:18:59.978641 11815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:18:59.980195 11815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:58545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:18:59.980429 11815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:18:59.980644 11815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:18:59.986313 11825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:59.986931 11822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:18:59.987170 11823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:18:59.988376 11815 server_base.cc:1034] running on GCE node
I20260925 13:18:59.988582 11815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:18:59.988847 11815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:18:59.990002 11815 hybrid_clock.cc:648] HybridClock initialized: now 1790342339989980 us; error 33 us; skew 500 ppm
Sep 25 13:18:59 dist-test-slave-t8d9 krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:18:59.992429 11815 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:18:59.992892 11815 webserver.cc:460] Webserver started at http://127.0.98.2:40479/ using document root <none> and password file <none>
I20260925 13:18:59.993130 11815 fs_manager.cc:359] Metadata directory not provided
I20260925 13:18:59.993181 11815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:18:59.993320 11815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:18:59 dist-test-slave-t8d9 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:19:00.014480 11815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "4e50ce2ad0f04edb9a65deb7c14094f6"
format_stamp: "Formatted at 2026-09-25 13:19:00 on dist-test-slave-t8d9"
server_key: "918200c7172bd73962b67f202d71c44d"
server_key_iv: "17101944ff21824e1c70f9329b06947d"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:00.014957 11815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4e50ce2ad0f04edb9a65deb7c14094f6"
format_stamp: "Formatted at 2026-09-25 13:19:00 on dist-test-slave-t8d9"
server_key: "918200c7172bd73962b67f202d71c44d"
server_key_iv: "17101944ff21824e1c70f9329b06947d"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:00.019562 11815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 13:19:00.033984 11832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:00.036510 11815 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 13:19:00.036609 11815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "4e50ce2ad0f04edb9a65deb7c14094f6"
format_stamp: "Formatted at 2026-09-25 13:19:00 on dist-test-slave-t8d9"
server_key: "918200c7172bd73962b67f202d71c44d"
server_key_iv: "17101944ff21824e1c70f9329b06947d"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:00.036726 11815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:00.067428 11815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:00.070263 11815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:00.070477 11815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:00.070815 11815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:00.071249 11815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:19:00.071341 11815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:00.071409 11815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:19:00.071463 11815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:00.079717 11815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34217
I20260925 13:19:00.080257 11815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:19:00.082069 11946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34217 every 8 connection(s)
I20260925 13:19:00.087793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 11815
I20260925 13:19:00.087904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:19:00.088132 392 external_mini_cluster.cc:1428] Setting key bba82aed3d01fd13489c550a075bee67
Sep 25 13:19:00 dist-test-slave-t8d9 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:00.094304 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.083255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42709 (local address 127.0.98.62:34609)
0925 13:19:00.083320 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:00.083322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:00.083581 (+ 259us) server_negotiation.cc:407] Connection header received
0925 13:19:00.084311 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:00.084313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:00.084325 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:00.084357 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:00.085246 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.086021 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.087487 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.087794 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.087834 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.090928 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:00.090930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:00.090930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:00.090938 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:00.092581 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:00.093084 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:00.093086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:00.093086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:00.093149 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:00.093451 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:00.093452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:00.093452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:00.093630 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:00.093675 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:00.094097 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 13:19:00.094196 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 13:19:00.095227 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.082958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34609 (local address 127.0.98.2:42709)
0925 13:19:00.083444 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:00.083491 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:00.084171 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:00.084555 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:00.084560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:00.084825 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:00.085136 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:00.085142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.086135 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.086135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:00.087386 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:00.087389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.088064 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.088065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:00.088280 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.089066 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:00.089077 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:00.090817 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:00.092674 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:00.092681 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:00.092683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:00.092997 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:00.093220 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:00.093221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:00.093221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:00.093368 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:00.093733 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:00.093740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:00.093929 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 13:19:00.094803 (+ 874us) client_negotiation.cc:240] Negotiation successful
0925 13:19:00.094981 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":87,"threads_started":1}
I20260925 13:19:00.095883 11947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34609
I20260925 13:19:00.095978 11947 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:00.096179 11947 heartbeater.cc:511] Master 127.0.98.62:34609 requested a full tablet report, sending...
I20260925 13:19:00.096632 11597 ts_manager.cc:194] Registered new tserver with Master: 4e50ce2ad0f04edb9a65deb7c14094f6 (127.0.98.2:34217)
I20260925 13:19:00.097158 11597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42709
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:19:00.129873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34609
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:19:00.256108 11952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:00.256542 11952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:00.256629 11952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:00.258522 11952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:00.258651 11952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:00.258728 11952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:19:00.260944 11952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:58545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:00.261461 11952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:00.261811 11952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:00.268268 11960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:00.268540 11957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:00.269356 11952 server_base.cc:1034] running on GCE node
W20260925 13:19:00.276100 11958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:00.276571 11952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:00.276906 11952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:00.278066 11952 hybrid_clock.cc:648] HybridClock initialized: now 1790342340278042 us; error 41 us; skew 500 ppm
Sep 25 13:19:00 dist-test-slave-t8d9 krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:00.281378 11952 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:19:00.281975 11952 webserver.cc:460] Webserver started at http://127.0.98.3:42957/ using document root <none> and password file <none>
I20260925 13:19:00.282246 11952 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:00.282307 11952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:00.282441 11952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:19:00 dist-test-slave-t8d9 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:19:00.296345 11952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4"
format_stamp: "Formatted at 2026-09-25 13:19:00 on dist-test-slave-t8d9"
server_key: "cfb0da0a9807929e95cbc15b91d24518"
server_key_iv: "ae1837177933f5a8efc10b3548c84d25"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:00.296919 11952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4"
format_stamp: "Formatted at 2026-09-25 13:19:00 on dist-test-slave-t8d9"
server_key: "cfb0da0a9807929e95cbc15b91d24518"
server_key_iv: "ae1837177933f5a8efc10b3548c84d25"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:00.301990 11952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.001s
I20260925 13:19:00.315487 11967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:00.315742 11952 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:19:00.315838 11952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4"
format_stamp: "Formatted at 2026-09-25 13:19:00 on dist-test-slave-t8d9"
server_key: "cfb0da0a9807929e95cbc15b91d24518"
server_key_iv: "ae1837177933f5a8efc10b3548c84d25"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:00.315932 11952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:00.347074 11952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:00.349766 11952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:00.349992 11952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:00.350306 11952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:00.350800 11952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:19:00.350894 11952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:00.350965 11952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:19:00.351019 11952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:00.358058 11952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43765
I20260925 13:19:00.358593 11952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:19:00.361749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 11952
I20260925 13:19:00.361851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:19:00.362053 392 external_mini_cluster.cc:1428] Setting key e59af020b22db8b4bfe1eb71bbf86f32
I20260925 13:19:00.365521 12080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43765 every 8 connection(s)
Sep 25 13:19:00 dist-test-slave-t8d9 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:00.375888 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.363776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42739 (local address 127.0.98.62:34609)
0925 13:19:00.363818 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:00.363820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:00.364020 (+ 200us) server_negotiation.cc:407] Connection header received
0925 13:19:00.365763 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:00.365765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:00.365779 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:00.365811 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:00.366735 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.367506 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.368993 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.369302 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.369346 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.372043 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:00.372045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:00.372045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:00.372053 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:00.373707 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:00.374719 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:00.374721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:00.374721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:00.374785 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:00.375090 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:00.375091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:00.375091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:00.375239 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:00.375282 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:00.375708 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 13:19:00.375793 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:19:00.376678 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.363476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34609 (local address 127.0.98.3:42739)
0925 13:19:00.363889 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:00.363940 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:00.365268 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:00.366010 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:00.366015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:00.366269 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:00.366618 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:00.366625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.367620 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.367621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:00.368890 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:00.368893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.369436 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.369437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:00.369606 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.370304 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:00.370315 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:00.371928 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:00.374041 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:00.374297 (+ 256us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:00.374302 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:00.374617 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:00.374857 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:00.374858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:00.374858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:00.375010 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:00.375330 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:00.375337 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:00.375539 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 13:19:00.376246 (+ 707us) client_negotiation.cc:240] Negotiation successful
0925 13:19:00.376432 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":94,"threads_started":1}
I20260925 13:19:00.377341 12081 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34609
I20260925 13:19:00.377434 12081 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:00.377646 12081 heartbeater.cc:511] Master 127.0.98.62:34609 requested a full tablet report, sending...
I20260925 13:19:00.378073 11597 ts_manager.cc:194] Registered new tserver with Master: d7da73d2b8ad4f85bb1b2a6828bcadb4 (127.0.98.3:43765)
I20260925 13:19:00.378552 11597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42739
I20260925 13:19:00.385124 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:19:00.397118 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.388870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.0.98.62:34609)
0925 13:19:00.388919 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:00.388921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:00.389252 (+ 331us) server_negotiation.cc:407] Connection header received
0925 13:19:00.389353 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:00.389354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:00.389367 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:00.389396 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:00.390227 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.391009 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.392124 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.392410 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.392456 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.394111 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:00.394113 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:00.394113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:00.394122 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:00.395769 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:00.396207 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:00.396209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:00.396209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:00.396273 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:00.396478 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:00.396479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:00.396479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:00.396620 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:00.396708 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:00.396928 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 13:19:00.397014 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:19:00.399657 11597 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:19:00.402168 11597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:19:00.412438 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.406037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37195 (local address 127.0.0.1:34960)
0925 13:19:00.407216 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:00.407263 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:00.407337 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:00.407639 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:00.407642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:00.407645 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:00.407743 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:00.407746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.409309 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.409310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:00.410639 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:00.410643 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.412094 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.412095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:00.412195 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.412211 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 13:19:00.412279 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 13:19:00.412340 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"thread_start_us":81,"threads_started":1}
I20260925 13:19:00.413133 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.406138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34960 (local address 127.0.98.1:37195)
0925 13:19:00.407147 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:00.407153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:00.407423 (+ 270us) server_negotiation.cc:407] Connection header received
0925 13:19:00.407464 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:00.407467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:00.407484 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:00.407553 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:00.407824 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.409221 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.410760 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.411922 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.411995 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.412058 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:00.413015 (+ 957us) server_negotiation.cc:299] Negotiation successful
0925 13:19:00.413063 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":58,"threads_started":1}
I20260925 13:19:00.414146 11743 tablet_service.cc:1467] Processing CreateTablet for tablet cf0be59c903a41b8828a8d865c9d2f69 (DEFAULT_TABLE table=test-table [id=7230207058a84c579fc1fee7826afff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:19:00.414562 11743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0be59c903a41b8828a8d865c9d2f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:00.422052 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.406268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34217 (local address 127.0.0.1:55136)
0925 13:19:00.412896 (+ 6628us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:00.412906 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:00.412940 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:00.413920 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:00.413922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:00.413923 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:00.413961 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:00.413962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.415317 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.415319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:00.416432 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:00.416435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.421751 (+ 5316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.421753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:00.421833 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.421837 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:19:00.421906 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 13:19:00.421962 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6615,"thread_start_us":73,"threads_started":1}
I20260925 13:19:00.424360 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.406451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43765 (local address 127.0.0.1:41782)
0925 13:19:00.416560 (+ 10109us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:00.416572 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:00.416619 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:00.416951 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:00.416952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:00.416955 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:00.417000 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:00.417002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.418309 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.418310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:00.419697 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:00.419701 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.424064 (+ 4363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.424067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:00.424146 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.424152 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 13:19:00.424210 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 13:19:00.424265 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10089,"thread_start_us":78,"threads_started":1}
I20260925 13:19:00.424978 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.406875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.0.98.3:43765)
0925 13:19:00.414097 (+ 7222us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:00.414101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:00.416682 (+ 2581us) server_negotiation.cc:407] Connection header received
0925 13:19:00.416747 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:00.416759 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:00.416779 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:00.416853 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:00.417074 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.418204 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.422529 (+ 4325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.423832 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.423923 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.424021 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:00.424802 (+ 781us) server_negotiation.cc:299] Negotiation successful
0925 13:19:00.424861 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7183,"thread_start_us":64,"threads_started":1}
I20260925 13:19:00.425995 12015 tablet_service.cc:1467] Processing CreateTablet for tablet cf0be59c903a41b8828a8d865c9d2f69 (DEFAULT_TABLE table=test-table [id=7230207058a84c579fc1fee7826afff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:19:00.426481 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0be59c903a41b8828a8d865c9d2f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:00.431180 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.406680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.0.98.2:34217)
0925 13:19:00.413662 (+ 6982us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:00.413667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:00.413678 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:19:00.413717 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:00.413719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:00.413736 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:00.413835 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:00.414180 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.415220 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.419914 (+ 4694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.421155 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.421422 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.421675 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:00.431005 (+ 9330us) server_negotiation.cc:299] Negotiation successful
0925 13:19:00.431078 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6938,"thread_start_us":98,"threads_started":1}
I20260925 13:19:00.431404 12100 tablet_bootstrap.cc:492] T cf0be59c903a41b8828a8d865c9d2f69 P 035fc3b36aac45dca47a47acbd0363f7: Bootstrap starting.
I20260925 13:19:00.432096 11881 tablet_service.cc:1467] Processing CreateTablet for tablet cf0be59c903a41b8828a8d865c9d2f69 (DEFAULT_TABLE table=test-table [id=7230207058a84c579fc1fee7826afff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:19:00.432283 12100 tablet_bootstrap.cc:654] T cf0be59c903a41b8828a8d865c9d2f69 P 035fc3b36aac45dca47a47acbd0363f7: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:00.432473 11881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0be59c903a41b8828a8d865c9d2f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:00.432590 12100 log.cc:824] T cf0be59c903a41b8828a8d865c9d2f69 P 035fc3b36aac45dca47a47acbd0363f7: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:00.432740 12101 tablet_bootstrap.cc:492] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4: Bootstrap starting.
I20260925 13:19:00.433732 12101 tablet_bootstrap.cc:654] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:00.434074 12101 log.cc:824] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:00.440330 12104 tablet_bootstrap.cc:492] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6: Bootstrap starting.
I20260925 13:19:00.446044 12104 tablet_bootstrap.cc:654] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:00.446444 12100 tablet_bootstrap.cc:492] T cf0be59c903a41b8828a8d865c9d2f69 P 035fc3b36aac45dca47a47acbd0363f7: No bootstrap required, opened a new log
I20260925 13:19:00.446508 12104 log.cc:824] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:00.446532 12100 ts_tablet_manager.cc:1397] T cf0be59c903a41b8828a8d865c9d2f69 P 035fc3b36aac45dca47a47acbd0363f7: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260925 13:19:00.446630 12101 tablet_bootstrap.cc:492] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4: No bootstrap required, opened a new log
I20260925 13:19:00.446710 12101 ts_tablet_manager.cc:1397] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260925 13:19:00.449224 12101 raft_consensus.cc:348] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035fc3b36aac45dca47a47acbd0363f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "4e50ce2ad0f04edb9a65deb7c14094f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34217 } } peers { permanent_uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43765 } }
I20260925 13:19:00.449792 12100 raft_consensus.cc:348] T cf0be59c903a41b8828a8d865c9d2f69 P 035fc3b36aac45dca47a47acbd0363f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035fc3b36aac45dca47a47acbd0363f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "4e50ce2ad0f04edb9a65deb7c14094f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34217 } } peers { permanent_uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43765 } }
I20260925 13:19:00.450076 12100 raft_consensus.cc:374] T cf0be59c903a41b8828a8d865c9d2f69 P 035fc3b36aac45dca47a47acbd0363f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:00.450176 12100 raft_consensus.cc:729] T cf0be59c903a41b8828a8d865c9d2f69 P 035fc3b36aac45dca47a47acbd0363f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035fc3b36aac45dca47a47acbd0363f7, State: Initialized, Role: FOLLOWER
I20260925 13:19:00.450434 12100 consensus_queue.cc:260] T cf0be59c903a41b8828a8d865c9d2f69 P 035fc3b36aac45dca47a47acbd0363f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035fc3b36aac45dca47a47acbd0363f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "4e50ce2ad0f04edb9a65deb7c14094f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34217 } } peers { permanent_uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43765 } }
I20260925 13:19:00.450871 12100 ts_tablet_manager.cc:1428] T cf0be59c903a41b8828a8d865c9d2f69 P 035fc3b36aac45dca47a47acbd0363f7: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260925 13:19:00.451045 11809 heartbeater.cc:503] Master 127.0.98.62:34609 was elected leader, sending a full tablet report...
I20260925 13:19:00.453619 12101 raft_consensus.cc:374] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:00.453778 12101 raft_consensus.cc:729] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7da73d2b8ad4f85bb1b2a6828bcadb4, State: Initialized, Role: FOLLOWER
I20260925 13:19:00.453985 12101 consensus_queue.cc:260] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035fc3b36aac45dca47a47acbd0363f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "4e50ce2ad0f04edb9a65deb7c14094f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34217 } } peers { permanent_uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43765 } }
I20260925 13:19:00.454411 12101 ts_tablet_manager.cc:1428] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260925 13:19:00.454581 12081 heartbeater.cc:503] Master 127.0.98.62:34609 was elected leader, sending a full tablet report...
I20260925 13:19:00.458869 12104 tablet_bootstrap.cc:492] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6: No bootstrap required, opened a new log
I20260925 13:19:00.458961 12104 ts_tablet_manager.cc:1397] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20260925 13:19:00.461081 12104 raft_consensus.cc:348] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035fc3b36aac45dca47a47acbd0363f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "4e50ce2ad0f04edb9a65deb7c14094f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34217 } } peers { permanent_uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43765 } }
I20260925 13:19:00.461285 12104 raft_consensus.cc:374] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:00.461319 12104 raft_consensus.cc:729] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e50ce2ad0f04edb9a65deb7c14094f6, State: Initialized, Role: FOLLOWER
I20260925 13:19:00.461629 12104 consensus_queue.cc:260] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035fc3b36aac45dca47a47acbd0363f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "4e50ce2ad0f04edb9a65deb7c14094f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34217 } } peers { permanent_uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43765 } }
I20260925 13:19:00.462615 12104 ts_tablet_manager.cc:1428] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 13:19:00.462787 11947 heartbeater.cc:503] Master 127.0.98.62:34609 was elected leader, sending a full tablet report...
Sep 25 13:19:00 dist-test-slave-t8d9 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260925 13:19:00.575856 11810 tablet.cc:2369] T cf0be59c903a41b8828a8d865c9d2f69 P 035fc3b36aac45dca47a47acbd0363f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:19:00.586092 11948 tablet.cc:2369] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:19:00.613161 12082 tablet.cc:2369] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:19:00.647591 12108 raft_consensus.cc:482] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:19:00.647744 12108 raft_consensus.cc:504] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035fc3b36aac45dca47a47acbd0363f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "4e50ce2ad0f04edb9a65deb7c14094f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34217 } } peers { permanent_uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43765 } }
I20260925 13:19:00.648126 12108 leader_election.cc:288] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 035fc3b36aac45dca47a47acbd0363f7 (127.0.98.1:37195), d7da73d2b8ad4f85bb1b2a6828bcadb4 (127.0.98.3:43765)
I20260925 13:19:00.655115 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.648769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44411 (local address 127.0.98.1:37195)
0925 13:19:00.648873 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:00.648875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:00.649062 (+ 187us) server_negotiation.cc:407] Connection header received
0925 13:19:00.649158 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:00.649159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:00.649170 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:00.649210 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:00.649415 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.650171 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.654103 (+ 3932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.654717 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.654767 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.654780 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:00.654989 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 13:19:00.655035 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 13:19:00.655495 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.648381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37195 (local address 127.0.98.2:44411)
0925 13:19:00.648994 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:00.649005 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:00.649092 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:00.649281 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:00.649282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:00.649284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:00.649349 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:00.649352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.650293 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.650294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:00.651518 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:00.651522 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.654840 (+ 3318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.654841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:00.654910 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.654914 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:19:00.655350 (+ 436us) client_negotiation.cc:240] Negotiation successful
0925 13:19:00.655407 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":95,"threads_started":1}
I20260925 13:19:00.655999 11751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0be59c903a41b8828a8d865c9d2f69" candidate_uuid: "4e50ce2ad0f04edb9a65deb7c14094f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035fc3b36aac45dca47a47acbd0363f7" is_pre_election: true
I20260925 13:19:00.656170 11751 raft_consensus.cc:2454] T cf0be59c903a41b8828a8d865c9d2f69 P 035fc3b36aac45dca47a47acbd0363f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e50ce2ad0f04edb9a65deb7c14094f6 in term 0.
I20260925 13:19:00.656451 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.648815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36267 (local address 127.0.98.3:43765)
0925 13:19:00.648881 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:00.648884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:00.651698 (+ 2814us) server_negotiation.cc:407] Connection header received
0925 13:19:00.651798 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:00.651799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:00.651813 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:00.651842 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:00.652035 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.652700 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.655508 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.656062 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.656107 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.656116 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:00.656343 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 13:19:00.656386 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 13:19:00.656586 11833 leader_election.cc:302] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035fc3b36aac45dca47a47acbd0363f7, 4e50ce2ad0f04edb9a65deb7c14094f6; no voters:
I20260925 13:19:00.656749 12108 raft_consensus.cc:2781] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:19:00.656817 12108 raft_consensus.cc:482] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:19:00.656840 12108 raft_consensus.cc:3037] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:00.656762 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.648589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43765 (local address 127.0.98.2:36267)
0925 13:19:00.651637 (+ 3048us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:00.651649 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:00.651727 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:00.651899 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:00.651900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:00.651901 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:00.651971 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:00.651973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.652822 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.652823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:00.653987 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:00.653990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.656175 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.656176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:00.656253 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.656257 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:19:00.656531 (+ 274us) client_negotiation.cc:240] Negotiation successful
0925 13:19:00.656577 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3030,"thread_start_us":39,"threads_started":1}
I20260925 13:19:00.657506 12029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0be59c903a41b8828a8d865c9d2f69" candidate_uuid: "4e50ce2ad0f04edb9a65deb7c14094f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4" is_pre_election: true
I20260925 13:19:00.657687 12029 raft_consensus.cc:2454] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e50ce2ad0f04edb9a65deb7c14094f6 in term 0.
I20260925 13:19:00.657843 12108 raft_consensus.cc:504] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035fc3b36aac45dca47a47acbd0363f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "4e50ce2ad0f04edb9a65deb7c14094f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34217 } } peers { permanent_uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43765 } }
I20260925 13:19:00.657979 12108 leader_election.cc:288] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [CANDIDATE]: Term 1 election: Requested vote from peers 035fc3b36aac45dca47a47acbd0363f7 (127.0.98.1:37195), d7da73d2b8ad4f85bb1b2a6828bcadb4 (127.0.98.3:43765)
I20260925 13:19:00.658149 12029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0be59c903a41b8828a8d865c9d2f69" candidate_uuid: "4e50ce2ad0f04edb9a65deb7c14094f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4"
I20260925 13:19:00.658218 12029 raft_consensus.cc:3037] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:00.658250 11751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0be59c903a41b8828a8d865c9d2f69" candidate_uuid: "4e50ce2ad0f04edb9a65deb7c14094f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035fc3b36aac45dca47a47acbd0363f7"
I20260925 13:19:00.658330 11751 raft_consensus.cc:3037] T cf0be59c903a41b8828a8d865c9d2f69 P 035fc3b36aac45dca47a47acbd0363f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:00.659034 12029 raft_consensus.cc:2454] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e50ce2ad0f04edb9a65deb7c14094f6 in term 1.
I20260925 13:19:00.659133 11751 raft_consensus.cc:2454] T cf0be59c903a41b8828a8d865c9d2f69 P 035fc3b36aac45dca47a47acbd0363f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e50ce2ad0f04edb9a65deb7c14094f6 in term 1.
I20260925 13:19:00.659243 11835 leader_election.cc:302] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e50ce2ad0f04edb9a65deb7c14094f6, d7da73d2b8ad4f85bb1b2a6828bcadb4; no voters:
I20260925 13:19:00.659379 12108 raft_consensus.cc:2781] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:19:00.659535 12108 raft_consensus.cc:686] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [term 1 LEADER]: Becoming Leader. State: Replica: 4e50ce2ad0f04edb9a65deb7c14094f6, State: Running, Role: LEADER
I20260925 13:19:00.659622 12108 consensus_queue.cc:237] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035fc3b36aac45dca47a47acbd0363f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } } peers { permanent_uuid: "4e50ce2ad0f04edb9a65deb7c14094f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34217 } } peers { permanent_uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43765 } }
I20260925 13:19:00.660451 11597 catalog_manager.cc:5495] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e50ce2ad0f04edb9a65deb7c14094f6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4e50ce2ad0f04edb9a65deb7c14094f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035fc3b36aac45dca47a47acbd0363f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e50ce2ad0f04edb9a65deb7c14094f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43765 } health_report { overall_health: UNKNOWN } } }
I20260925 13:19:00.701422 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.698269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.0.98.2:34217)
0925 13:19:00.698337 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:00.698339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:00.698349 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:19:00.698389 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:00.698390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:00.698415 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:00.698488 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:19:00.698636 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.699347 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.700468 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:00.700777 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:00.700853 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:00.701033 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:19:00.701171 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:19:00.701249 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:00.701304 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 13:19:00.701327 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 13:19:00.703538 12029 raft_consensus.cc:1261] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4 [term 1 FOLLOWER]: Refusing update from remote peer 4e50ce2ad0f04edb9a65deb7c14094f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:19:00.703562 11751 raft_consensus.cc:1261] T cf0be59c903a41b8828a8d865c9d2f69 P 035fc3b36aac45dca47a47acbd0363f7 [term 1 FOLLOWER]: Refusing update from remote peer 4e50ce2ad0f04edb9a65deb7c14094f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:19:00.703912 12108 consensus_queue.cc:1035] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "035fc3b36aac45dca47a47acbd0363f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:19:00.704087 12108 consensus_queue.cc:1035] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7da73d2b8ad4f85bb1b2a6828bcadb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:19:00.708681 12114 mvcc.cc:205] Tried to move back new op lower bound from 7333242227518914560 to 7333242227342426112. Current Snapshot: MvccSnapshot[applied={T|T < 7333242227518914560 or (T in {7333242227518914560})}]
I20260925 13:19:00.709678 12116 mvcc.cc:205] Tried to move back new op lower bound from 7333242227518914560 to 7333242227342426112. Current Snapshot: MvccSnapshot[applied={T|T < 7333242227518914560}]
I20260925 13:19:00.709887 11596 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:19:00.709986 11596 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:19:00.710486 12115 mvcc.cc:205] Tried to move back new op lower bound from 7333242227518914560 to 7333242227342426112. Current Snapshot: MvccSnapshot[applied={T|T < 7333242227518914560 or (T in {7333242227518914560})}]
I20260925 13:19:00.710714 11596 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9aa1222235264e8c8bdef7fd2fa9f373: Sending DeleteTablet for 3 replicas of tablet cf0be59c903a41b8828a8d865c9d2f69
I20260925 13:19:00.711302 12015 tablet_service.cc:1514] Processing DeleteTablet for tablet cf0be59c903a41b8828a8d865c9d2f69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:19:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41782
I20260925 13:19:00.711369 11743 tablet_service.cc:1514] Processing DeleteTablet for tablet cf0be59c903a41b8828a8d865c9d2f69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:19:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34960
I20260925 13:19:00.711467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 11678
I20260925 13:19:00.711573 11881 tablet_service.cc:1514] Processing DeleteTablet for tablet cf0be59c903a41b8828a8d865c9d2f69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:19:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55136
I20260925 13:19:00.717674 12126 tablet_replica.cc:330] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6: stopping tablet replica
I20260925 13:19:00.717801 12126 raft_consensus.cc:2229] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:19:00.717994 12126 raft_consensus.cc:2258] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:19:00.718194 12124 tablet_replica.cc:330] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4: stopping tablet replica
I20260925 13:19:00.718636 12126 ts_tablet_manager.cc:1905] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:19:00.720319 12126 ts_tablet_manager.cc:1918] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:19:00.720434 12126 log.cc:1196] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/cf0be59c903a41b8828a8d865c9d2f69
I20260925 13:19:00.720589 12124 raft_consensus.cc:2229] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:19:00.720949 12126 ts_tablet_manager.cc:1939] T cf0be59c903a41b8828a8d865c9d2f69 P 4e50ce2ad0f04edb9a65deb7c14094f6: Deleting consensus metadata
I20260925 13:19:00.721129 12124 raft_consensus.cc:2258] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:19:00.721659 11581 catalog_manager.cc:4841] TS 4e50ce2ad0f04edb9a65deb7c14094f6 (127.0.98.2:34217): tablet cf0be59c903a41b8828a8d865c9d2f69 (table test-table [id=7230207058a84c579fc1fee7826afff6]) successfully deleted
I20260925 13:19:00.722174 12124 ts_tablet_manager.cc:1905] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:19:00.723590 12124 ts_tablet_manager.cc:1918] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:19:00.723718 12124 log.cc:1196] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/cf0be59c903a41b8828a8d865c9d2f69
I20260925 13:19:00.724283 12124 ts_tablet_manager.cc:1939] T cf0be59c903a41b8828a8d865c9d2f69 P d7da73d2b8ad4f85bb1b2a6828bcadb4: Deleting consensus metadata
I20260925 13:19:00.724738 11583 catalog_manager.cc:4841] TS d7da73d2b8ad4f85bb1b2a6828bcadb4 (127.0.98.3:43765): tablet cf0be59c903a41b8828a8d865c9d2f69 (table test-table [id=7230207058a84c579fc1fee7826afff6]) successfully deleted
W20260925 13:19:00.727898 11581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37195 (error 108)
I20260925 13:19:00.728876 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:00.728751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37195 (local address 127.0.0.1:34970)
0925 13:19:00.728799 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:00.728836 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 13:19:00.728894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 11815
W20260925 13:19:00.729516 11581 catalog_manager.cc:4584] TS 035fc3b36aac45dca47a47acbd0363f7 (127.0.98.1:37195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf0be59c903a41b8828a8d865c9d2f69: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37195: connect: Connection refused (error 111)
I20260925 13:19:00.736470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 11952
I20260925 13:19:00.742808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 11565
I20260925 13:19:00.747797 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 13:19:01.082397 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 13:19:01.082499 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 13:19:01.417110 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 13:19:01.417 UTC [11151] LOG: received smart shutdown request
2026-09-25 13:19:01.420 UTC [11151] LOG: background worker "logical replication launcher" (PID 11160) exited with exit code 1
2026-09-25 13:19:01.420 UTC [11155] LOG: shutting down
2026-09-25 13:19:01.581 UTC [11151] LOG: database system is shut down
2026-09-25T13:19:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45523 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:19:01 dist-test-slave-t8d9 krb5kdc[12136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:19:01 dist-test-slave-t8d9 krb5kdc[12136](info): set up 2 sockets
Sep 25 13:19:01 dist-test-slave-t8d9 krb5kdc[12136](info): commencing operation
krb5kdc: starting...
W20260925 13:19:02.621188 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:19:02 dist-test-slave-t8d9 krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:19:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:19:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 13:19:02.780273 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskC7bpD2/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/postgres -l logfile start
2026-09-25 13:19:05.083 UTC [12166] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 13:19:05.083 UTC [12166] LOG: listening on IPv4 address "127.0.98.20", port 41661
2026-09-25 13:19:05.086 UTC [12166] LOG: listening on Unix socket "/tmp/.s.PGSQL.41661"
2026-09-25 13:19:05.113 UTC [12169] LOG: database system was shut down at 2026-09-25 13:19:03 UTC
2026-09-25 13:19:05.117 UTC [12166] LOG: database system is ready to accept connections
I20260925 13:19:05.981052 392 mini_postgres.cc:96] Postgres bound to 41661
2026-09-25 13:19:05.986 UTC [12177] FATAL: database "slave" does not exist
127.0.98.20:41661 - accepting connections
I20260925 13:19:05.986991 392 mini_ranger.cc:161] Starting Ranger...
I20260925 13:19:06.001400 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 13:19:06.457520 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 13:19:06.457617 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-admin
2026-09-25 13:19:06,719 [I] DB FLAVOR :POSTGRES
2026-09-25 13:19:06,720 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 13:19:06,720 [I] Checking connection..
2026-09-25 13:19:06,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 13:19:06,977 [I] Checking connection passed.
2026-09-25 13:19:06,977 [I] --------- Verifying version history table ---------
2026-09-25 13:19:06,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 13:19:07,205 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 13:19:07,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 13:19:07,468 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 13:19:07,468 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 13:19:07,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 13:19:07.708 UTC [12241] WARNING: there is no transaction in progress
2026-09-25 13:19:07,714 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 13:19:07,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 13:19:07,942 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 13:19:07,943 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 13:19:07,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 13:19:08,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 13:19:08,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-t8d9.c.gcp-upstream.internal','N') ;"
2026-09-25 13:19:08,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 13:19:08,944 [I] Table x_portal_user does not exist in database ranger
2026-09-25 13:19:08,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 13:19:09,204 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 13:19:09,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 13:19:09,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 13:19:09,675 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 13:19:09,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 13:19:10.644 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:10.661 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:10.678 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:10.693 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:10.709 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:10.725 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:10.781 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:10.788 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:10.803 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:10.813 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:10.823 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:10.837 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:10.847 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:10.853 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:10.860 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:10.868 UTC [12349] WARNING: there is no transaction in progress
2026-09-25 13:19:11,476 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 13:19:11,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 13:19:11,710 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 13:19:11,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 13:19:11,969 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 13:19:11,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 13:19:12,209 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 13:19:12,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 13:19:12,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 13:19:12,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-t8d9.c.gcp-upstream.internal';"
2026-09-25 13:19:12,984 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 13:19:12,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 13:19:13,211 [I] DB_PATCHES have already been applied
I20260925 13:19:13.216563 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskC7bpD2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 13:19:13.216626 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 13:19:13.219419 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:55307
Sep 25, 2026 1:19:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:19:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:19:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 1:19:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 1:19:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 1:19:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 1:19:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 13:19:14.218321 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.231s user 0.007s sys 0.001s
Sep 25, 2026 1:19:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 1:19:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 1:19:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 1:19:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 1:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 1:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 1:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 1:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 1:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 1:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 1:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 1:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 1:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 1:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 1:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 1:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 1:19:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 1:19:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 13:19:34.065124 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 13:19:34 dist-test-slave-t8d9 krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342374, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 13:19:34.138109 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 13:19:34.146407 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 13:19:34.945776 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 13:19:34.945860 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-kms
I20260925 13:19:34.945878 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41661
2026-09-25 13:19:35,101 [I] DB FLAVOR :POSTGRES
2026-09-25 13:19:35,102 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 13:19:35,102 [I] Checking connection
2026-09-25 13:19:35,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 13:19:35,356 [I] connection success
2026-09-25 13:19:35,356 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 13:19:35,357 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 13:19:35,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 13:19:35,603 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 13:19:35,603 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 13:19:35,603 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 13:19:35,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41661/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 13:19:35,846 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 13:19:36.265522 392 mini_ranger_kms.cc:310] Created kms service
I20260925 13:19:36.445761 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 13:19:36.486438 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 13:19:36 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342374, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:19:36.865579 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 13:19:36.865697 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskC7bpD2/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskC7bpD2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 13:19:36.865716 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 13:19:36.868811 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12529
I20260925 13:19:36.868896 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:43489
Sep 25, 2026 1:19:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:19:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:19:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 1:19:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 1:19:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 1:19:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 13:19:38.872701 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.735s user 0.006s sys 0.132s
I20260925 13:19:38.872808 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 13:19:38.872853 392 mini_ranger_kms.cc:401] 127.0.98.20:43489/kms/v1/keys
Sep 25 13:19:39 dist-test-slave-t8d9 krb5kdc[12136](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790342379, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 13:19:39 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342379, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 13:19:39 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342379, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 13:19:42 dist-test-slave-t8d9 krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:19:42.160219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44959
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44959
--ranger_config_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43489/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:19:42.240052 12580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:42.240263 12580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:42.240293 12580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:42.241452 12580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:19:42.241496 12580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:42.241514 12580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:42.241555 12580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:19:42.241573 12580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:19:42.243189 12580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:43489/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44959
--ranger_config_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:42.243407 12580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:42.243633 12580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:42.248238 12586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:42.248278 12585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:42.248245 12588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:42.248948 12580 server_base.cc:1034] running on GCE node
I20260925 13:19:42.249207 12580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:42.249482 12580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:42.250669 12580 hybrid_clock.cc:648] HybridClock initialized: now 1790342382250656 us; error 33 us; skew 500 ppm
Sep 25 13:19:42 dist-test-slave-t8d9 krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:42.253070 12580 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:19:42.253511 12580 webserver.cc:460] Webserver started at http://127.0.98.62:42473/ using document root <none> and password file <none>
I20260925 13:19:42.253736 12580 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:42.253778 12580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:42.253898 12580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:19:42 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:19:42.315259 12580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "2bc95a2806a04e1b8157a65e295aec92"
format_stamp: "Formatted at 2026-09-25 13:19:42 on dist-test-slave-t8d9"
server_key: "000f60f6b74b7b76bed80ac9f07c4721"
server_key_iv: "5dcbb78c8a7d9d3e5c8a529e9814ec0f"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:42.315716 12580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "2bc95a2806a04e1b8157a65e295aec92"
format_stamp: "Formatted at 2026-09-25 13:19:42 on dist-test-slave-t8d9"
server_key: "000f60f6b74b7b76bed80ac9f07c4721"
server_key_iv: "5dcbb78c8a7d9d3e5c8a529e9814ec0f"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:42.317149 12580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:19:42.332743 12595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:42.332947 12580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:19:42.333069 12580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "2bc95a2806a04e1b8157a65e295aec92"
format_stamp: "Formatted at 2026-09-25 13:19:42 on dist-test-slave-t8d9"
server_key: "000f60f6b74b7b76bed80ac9f07c4721"
server_key_iv: "5dcbb78c8a7d9d3e5c8a529e9814ec0f"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:42.333168 12580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:42.350065 12580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:42.351900 12580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:42.352058 12580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:42.356885 12580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44959
I20260925 13:19:42.356913 12647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44959 every 8 connection(s)
I20260925 13:19:42.357349 12580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:19:42.358045 12648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:42.360292 12648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92: Bootstrap starting.
I20260925 13:19:42.360862 12648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:42.361122 12648 log.cc:824] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:42.361800 12648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92: No bootstrap required, opened a new log
I20260925 13:19:42.363791 12648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc95a2806a04e1b8157a65e295aec92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260925 13:19:42.363936 12648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:42.363976 12648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc95a2806a04e1b8157a65e295aec92, State: Initialized, Role: FOLLOWER
I20260925 13:19:42.364117 12648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc95a2806a04e1b8157a65e295aec92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260925 13:19:42.364194 12648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:19:42.364224 12648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:19:42.364270 12648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:42.364789 12648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc95a2806a04e1b8157a65e295aec92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260925 13:19:42.364882 12648 leader_election.cc:302] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bc95a2806a04e1b8157a65e295aec92; no voters:
I20260925 13:19:42.365049 12648 leader_election.cc:288] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:19:42.365087 12651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:19:42.365221 12651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 1 LEADER]: Becoming Leader. State: Replica: 2bc95a2806a04e1b8157a65e295aec92, State: Running, Role: LEADER
I20260925 13:19:42.365319 12648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:19:42.365365 12651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc95a2806a04e1b8157a65e295aec92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260925 13:19:42.365696 12651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bc95a2806a04e1b8157a65e295aec92. Latest consensus state: current_term: 1 leader_uuid: "2bc95a2806a04e1b8157a65e295aec92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc95a2806a04e1b8157a65e295aec92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } } }
I20260925 13:19:42.365770 12651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [sys.catalog]: This master's current role is: LEADER
I20260925 13:19:42.366060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 12580
I20260925 13:19:42.366153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:19:42.366338 392 external_mini_cluster.cc:1428] Setting key 2a254adc9d61515c94f220e3da566d0b
I20260925 13:19:42.366782 12648 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 13:19:42.369071 12653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bc95a2806a04e1b8157a65e295aec92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc95a2806a04e1b8157a65e295aec92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } } }
I20260925 13:19:42.369278 12653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [sys.catalog]: This master's current role is: LEADER
I20260925 13:19:42.369367 12657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:19:42.370445 12657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:19:42.378386 12657 catalog_manager.cc:1347] Generated new cluster ID: a5baf4195b7f46159286d8522be72527
I20260925 13:19:42.378446 12657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:19:42 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:42.382812 12667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:42.366811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.0.98.62:44959)
0925 13:19:42.369901 (+ 3090us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:42.369931 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:42.370000 (+ 69us) server_negotiation.cc:407] Connection header received
0925 13:19:42.370269 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:42.370276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:42.370351 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:42.370519 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:42.372933 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:42.374389 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:42.375431 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:42.376056 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:42.376154 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:42.379048 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:42.379052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:42.379053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:42.379065 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:42.380908 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:42.381275 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:42.381278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:42.381278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:42.381366 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:42.381563 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:42.381563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:42.381564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:42.381854 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:42.381999 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:42.382407 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 13:19:42.382521 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2956,"thread_start_us":79,"threads_started":1}
I20260925 13:19:42.405439 12657 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:19:42.406237 12657 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:19:42.421200 12657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92: Generated new TSK 0
I20260925 13:19:42.421430 12657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:19:43.160724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44959
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:19:43.279827 12693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:43.280100 12693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:43.280140 12693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:43.281893 12693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:43.281950 12693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:43.281992 12693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:19:43.284097 12693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:43489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:43.284363 12693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:43.284693 12693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:43.290009 12701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:43.290032 12699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:43.290046 12698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:43.290575 12693 server_base.cc:1034] running on GCE node
I20260925 13:19:43.290783 12693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:43.291059 12693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:43.292235 12693 hybrid_clock.cc:648] HybridClock initialized: now 1790342383292216 us; error 31 us; skew 500 ppm
Sep 25 13:19:43 dist-test-slave-t8d9 krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:43.295363 12693 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:19:43.295974 12693 webserver.cc:460] Webserver started at http://127.0.98.1:35185/ using document root <none> and password file <none>
I20260925 13:19:43.296263 12693 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:43.296324 12693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:43.296459 12693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:19:43 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:19:43.315241 12693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "3304e037bbb1406f929346d0d599d0d3"
format_stamp: "Formatted at 2026-09-25 13:19:43 on dist-test-slave-t8d9"
server_key: "915db16e03a88765103cf9d400748a14"
server_key_iv: "3530aba94da0845263d03aa4eaa79f64"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:43.315707 12693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3304e037bbb1406f929346d0d599d0d3"
format_stamp: "Formatted at 2026-09-25 13:19:43 on dist-test-slave-t8d9"
server_key: "915db16e03a88765103cf9d400748a14"
server_key_iv: "3530aba94da0845263d03aa4eaa79f64"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:43.323163 12693 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
I20260925 13:19:43.335412 12710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:43.335587 12693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:19:43.335672 12693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "3304e037bbb1406f929346d0d599d0d3"
format_stamp: "Formatted at 2026-09-25 13:19:43 on dist-test-slave-t8d9"
server_key: "915db16e03a88765103cf9d400748a14"
server_key_iv: "3530aba94da0845263d03aa4eaa79f64"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:43.335772 12693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:43.348338 12693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:43.350330 12693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:43.350474 12693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:43.350683 12693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:43.351029 12693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:19:43.351069 12693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:43.351109 12693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:19:43.351130 12693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:43.357728 12693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46219
I20260925 13:19:43.358150 12693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:19:43.359187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 12693
I20260925 13:19:43.359256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:19:43.359375 392 external_mini_cluster.cc:1428] Setting key bb779b442982ad4f3a16d3fe2a5ea03e
I20260925 13:19:43.361414 12823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46219 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 25 13:19:43 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:43.375983 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:43.362214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44959 (local address 127.0.98.1:46359)
0925 13:19:43.364830 (+ 2616us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:43.364882 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:43.365667 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:43.365932 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:43.365936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:43.366193 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:43.366396 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:43.366402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:43.367174 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:43.367175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:43.367977 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:43.367979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:43.368349 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:43.368350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:43.368484 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:43.368974 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:43.368983 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:43.372264 (+ 3281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:43.374259 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:43.374267 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:43.374269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:43.374607 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:43.374833 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:43.374834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:43.374834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:43.374970 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:43.375259 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:43.375266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:43.375454 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 13:19:43.375560 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 13:19:43.375735 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2565,"thread_start_us":86,"threads_started":1}
I20260925 13:19:43.376386 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:43.362229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46359 (local address 127.0.98.62:44959)
0925 13:19:43.362361 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:43.362364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:43.365006 (+ 2642us) server_negotiation.cc:407] Connection header received
0925 13:19:43.365771 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:43.365772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:43.365786 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:43.365852 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:43.366486 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:43.367060 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:43.368060 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:43.368262 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:43.368317 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:43.372395 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:43.372398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:43.372398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:43.372407 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:43.374150 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:43.374702 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:43.374703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:43.374703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:43.374766 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:43.375037 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:43.375037 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:43.375038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:43.375189 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:43.376152 (+ 963us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:43.376197 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 13:19:43.376283 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260925 13:19:43.376736 12824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44959
I20260925 13:19:43.376837 12824 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:43.377030 12824 heartbeater.cc:511] Master 127.0.98.62:44959 requested a full tablet report, sending...
I20260925 13:19:43.377626 12612 ts_manager.cc:194] Registered new tserver with Master: 3304e037bbb1406f929346d0d599d0d3 (127.0.98.1:46219)
I20260925 13:19:43.378541 12612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:19:43.388244 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44959
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 13:19:43 dist-test-slave-t8d9 krb5kdc[12136](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790342383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 13:19:43.502954 12830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:43.503211 12830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:43.503248 12830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:43.504909 12830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:43.504962 12830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:43.505003 12830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:19:43.507202 12830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:43489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:43.507483 12830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:43.507772 12830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:43.514119 12837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:43.514210 12838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:43.514400 12840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:43.514551 12830 server_base.cc:1034] running on GCE node
I20260925 13:19:43.514755 12830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:43.515010 12830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:43.516191 12830 hybrid_clock.cc:648] HybridClock initialized: now 1790342383516173 us; error 34 us; skew 500 ppm
Sep 25 13:19:43 dist-test-slave-t8d9 krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:43.518971 12830 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:19:43.519474 12830 webserver.cc:460] Webserver started at http://127.0.98.2:45063/ using document root <none> and password file <none>
I20260925 13:19:43.519704 12830 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:43.519752 12830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:43.519896 12830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:19:43 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:19:43.536198 12830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6"
format_stamp: "Formatted at 2026-09-25 13:19:43 on dist-test-slave-t8d9"
server_key: "8c1ed468bf4acfdbbe454cedbc2b7c75"
server_key_iv: "68c1ca2f2fa5c60cad0ea691b7f92d52"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:43.536651 12830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6"
format_stamp: "Formatted at 2026-09-25 13:19:43 on dist-test-slave-t8d9"
server_key: "8c1ed468bf4acfdbbe454cedbc2b7c75"
server_key_iv: "68c1ca2f2fa5c60cad0ea691b7f92d52"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:43.541594 12830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 13:19:43.553803 12847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:43.553995 12830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:19:43.554091 12830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6"
format_stamp: "Formatted at 2026-09-25 13:19:43 on dist-test-slave-t8d9"
server_key: "8c1ed468bf4acfdbbe454cedbc2b7c75"
server_key_iv: "68c1ca2f2fa5c60cad0ea691b7f92d52"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:43.554189 12830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:43.570207 12830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:43.572548 12830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:43.572718 12830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:43.572976 12830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:43.573343 12830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:19:43.573395 12830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:43.573432 12830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:19:43.573452 12830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:43.579521 12830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46553
I20260925 13:19:43.579957 12830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:19:43.581954 12960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46553 every 8 connection(s)
I20260925 13:19:43.584898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 12830
I20260925 13:19:43.584983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:19:43.585150 392 external_mini_cluster.cc:1428] Setting key a634fe429560e5f1946f66c79601565f
Sep 25 13:19:43 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:43.594462 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:43.583019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39887 (local address 127.0.98.62:44959)
0925 13:19:43.583108 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:43.583110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:43.583254 (+ 144us) server_negotiation.cc:407] Connection header received
0925 13:19:43.583942 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:43.583944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:43.583957 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:43.583988 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:43.584873 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:43.585716 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:43.588026 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:43.588339 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:43.588383 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:43.591141 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:43.591143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:43.591144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:43.591152 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:43.592816 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:43.593291 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:43.593293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:43.593293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:43.593352 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:43.593679 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:43.593681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:43.593681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:43.593822 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:43.593866 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:43.594252 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 13:19:43.594360 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 13:19:43.595274 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:43.582718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44959 (local address 127.0.98.2:39887)
0925 13:19:43.583108 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:43.583153 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:43.583815 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:43.584178 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:43.584183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:43.584436 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:43.584750 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:43.584762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:43.586483 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:43.586483 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:43.587927 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:43.587930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:43.588496 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:43.588497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:43.588641 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:43.589311 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:43.589332 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:43.591031 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:43.592910 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:43.592917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:43.592919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:43.593204 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:43.593418 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:43.593419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:43.593419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:43.593595 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:43.593911 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:43.593917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:43.594100 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 13:19:43.594850 (+ 750us) client_negotiation.cc:240] Negotiation successful
0925 13:19:43.595023 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":93,"threads_started":1}
I20260925 13:19:43.595930 12961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44959
I20260925 13:19:43.596071 12961 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:43.596302 12961 heartbeater.cc:511] Master 127.0.98.62:44959 requested a full tablet report, sending...
I20260925 13:19:43.596709 12612 ts_manager.cc:194] Registered new tserver with Master: e6dbca4bf9664fb2a33e47cd652ce7b6 (127.0.98.2:46553)
I20260925 13:19:43.597189 12612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39887
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:19:43.624639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44959
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:19:43.752701 12967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:43.753088 12967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:43.753180 12967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:43.755116 12967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:43.755252 12967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:43.755338 12967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:19:43.757731 12967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:43489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:43.758345 12967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:43.758713 12967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:43.764915 12975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:43.765267 12972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:43.765367 12967 server_base.cc:1034] running on GCE node
W20260925 13:19:43.765580 12973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:43.765830 12967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:43.766114 12967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:43.767272 12967 hybrid_clock.cc:648] HybridClock initialized: now 1790342383767247 us; error 36 us; skew 500 ppm
Sep 25 13:19:43 dist-test-slave-t8d9 krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:43.770280 12967 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:19:43.770848 12967 webserver.cc:460] Webserver started at http://127.0.98.3:41533/ using document root <none> and password file <none>
I20260925 13:19:43.771116 12967 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:43.771183 12967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:43.771317 12967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 13:19:43 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:19:43.784361 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "add6026d4645419c984394e87260f3c4"
format_stamp: "Formatted at 2026-09-25 13:19:43 on dist-test-slave-t8d9"
server_key: "a90489a6531a9ea76b03f5e7b4fddce5"
server_key_iv: "429c6a071a087792908018f314c6c627"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:43.784736 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "add6026d4645419c984394e87260f3c4"
format_stamp: "Formatted at 2026-09-25 13:19:43 on dist-test-slave-t8d9"
server_key: "a90489a6531a9ea76b03f5e7b4fddce5"
server_key_iv: "429c6a071a087792908018f314c6c627"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:43.786123 12967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:19:43.813339 12982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:43.813660 12967 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.000s
I20260925 13:19:43.813771 12967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "add6026d4645419c984394e87260f3c4"
format_stamp: "Formatted at 2026-09-25 13:19:43 on dist-test-slave-t8d9"
server_key: "a90489a6531a9ea76b03f5e7b4fddce5"
server_key_iv: "429c6a071a087792908018f314c6c627"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:43.813880 12967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:43.835798 12967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:43.838376 12967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:43.838595 12967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:43.838882 12967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:43.839318 12967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:19:43.839411 12967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:43.839486 12967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:19:43.839540 12967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:43.847182 12967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40225
I20260925 13:19:43.847661 12967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:19:43.848075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 12967
I20260925 13:19:43.848171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:19:43.848359 392 external_mini_cluster.cc:1428] Setting key 832ea38c7930b48d4129dfcd9ed7f6cf
I20260925 13:19:43.850827 13095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40225 every 8 connection(s)
Sep 25 13:19:43 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:43.872767 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:43.850216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44959 (local address 127.0.98.3:55947)
0925 13:19:43.861669 (+ 11453us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:43.861750 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:43.862451 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:43.862814 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:43.862820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:43.863105 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:43.863424 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:43.863431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:43.864417 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:43.864418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:43.865840 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:43.865842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:43.866389 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:43.866390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:43.866536 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:43.867216 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:43.867227 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:43.868865 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:43.870840 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:43.870847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:43.870850 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:43.871312 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:43.871546 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:43.871547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:43.871547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:43.871704 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:43.872005 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:43.872012 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:43.872203 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 13:19:43.872322 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 13:19:43.872503 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11381,"thread_start_us":87,"threads_started":1}
I20260925 13:19:43.873293 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:43.850488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55947 (local address 127.0.98.62:44959)
0925 13:19:43.850531 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:43.850533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:43.861853 (+ 11320us) server_negotiation.cc:407] Connection header received
0925 13:19:43.862580 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:43.862581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:43.862593 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:43.862658 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:43.863541 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:43.864313 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:43.865932 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:43.866232 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:43.866298 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:43.869001 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:43.869002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:43.869003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:43.869011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:43.870742 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:43.871408 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:43.871409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:43.871410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:43.871475 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:43.871775 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:43.871776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:43.871776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:43.871912 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:43.871966 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:43.873115 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0925 13:19:43.873201 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:19:43.874104 13096 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44959
I20260925 13:19:43.874259 13096 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:43.874498 13096 heartbeater.cc:511] Master 127.0.98.62:44959 requested a full tablet report, sending...
I20260925 13:19:43.874974 12612 ts_manager.cc:194] Registered new tserver with Master: add6026d4645419c984394e87260f3c4 (127.0.98.3:40225)
I20260925 13:19:43.875494 12612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55947
I20260925 13:19:43.881759 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:19:43.882529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 12693
I20260925 13:19:43.890684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 12830
I20260925 13:19:43.897039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 12967
I20260925 13:19:43.902966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 12580
I20260925 13:19:43.909029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44959
--webserver_interface=127.0.98.62
--webserver_port=42473
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44959
--ranger_config_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43489/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43489/kms with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:19:43.989619 13099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:43.989818 13099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:43.989852 13099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:43.990978 13099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:19:43.991016 13099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:43.991029 13099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:43.991039 13099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:19:43.991056 13099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:19:43.992558 13099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43489/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44959
--ranger_config_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:43.992734 13099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:43.992901 13099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:43.997377 13104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:43.997385 13105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:43.997394 13107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:43.997568 13099 server_base.cc:1034] running on GCE node
I20260925 13:19:43.997794 13099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:43.998066 13099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:43.999225 13099 hybrid_clock.cc:648] HybridClock initialized: now 1790342383999200 us; error 38 us; skew 500 ppm
Sep 25 13:19:43 dist-test-slave-t8d9 krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:44.001678 13099 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:19:44.002123 13099 webserver.cc:460] Webserver started at http://127.0.98.62:42473/ using document root <none> and password file <none>
I20260925 13:19:44.002342 13099 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:44.002398 13099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 13:19:44 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:19:44.046124 13099 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:19:44.049710 13115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:44.049948 13099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 13:19:44.050082 13099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "2bc95a2806a04e1b8157a65e295aec92"
format_stamp: "Formatted at 2026-09-25 13:19:42 on dist-test-slave-t8d9"
server_key: "000f60f6b74b7b76bed80ac9f07c4721"
server_key_iv: "5dcbb78c8a7d9d3e5c8a529e9814ec0f"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:44.050617 13099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:44.062945 13099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:44.064855 13099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:44.064982 13099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:44.069060 13099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44959
I20260925 13:19:44.069099 13167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44959 every 8 connection(s)
I20260925 13:19:44.069470 13099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:19:44.070740 13168 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:19:44.071389 13168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92: Bootstrap starting.
I20260925 13:19:44.074723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 13099
I20260925 13:19:44.075028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46219
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44959
--webserver_port=35185
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43489/kms with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:19:44.079579 13168 log.cc:824] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:44.081403 13168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:19:44.081713 13168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92: Bootstrap complete.
I20260925 13:19:44.083961 13168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc95a2806a04e1b8157a65e295aec92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260925 13:19:44.084206 13168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc95a2806a04e1b8157a65e295aec92, State: Initialized, Role: FOLLOWER
I20260925 13:19:44.084352 13168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc95a2806a04e1b8157a65e295aec92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260925 13:19:44.084416 13168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:19:44.084457 13168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:19:44.084504 13168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 1 FOLLOWER]: Advancing to term 2
I20260925 13:19:44.085633 13168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc95a2806a04e1b8157a65e295aec92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260925 13:19:44.085739 13168 leader_election.cc:302] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bc95a2806a04e1b8157a65e295aec92; no voters:
I20260925 13:19:44.085917 13168 leader_election.cc:288] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 13:19:44.085989 13172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 13:19:44.086151 13172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [term 2 LEADER]: Becoming Leader. State: Replica: 2bc95a2806a04e1b8157a65e295aec92, State: Running, Role: LEADER
I20260925 13:19:44.086189 13168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:19:44.086253 13172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc95a2806a04e1b8157a65e295aec92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260925 13:19:44.086578 13173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2bc95a2806a04e1b8157a65e295aec92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc95a2806a04e1b8157a65e295aec92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } } }
I20260925 13:19:44.086648 13173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [sys.catalog]: This master's current role is: LEADER
I20260925 13:19:44.086791 13174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bc95a2806a04e1b8157a65e295aec92. Latest consensus state: current_term: 2 leader_uuid: "2bc95a2806a04e1b8157a65e295aec92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc95a2806a04e1b8157a65e295aec92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } } }
I20260925 13:19:44.086836 13174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92 [sys.catalog]: This master's current role is: LEADER
I20260925 13:19:44.087221 13178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:19:44.087749 13178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:19:44.087869 13178 catalog_manager.cc:1259] Loaded cluster ID: a5baf4195b7f46159286d8522be72527
I20260925 13:19:44.087895 13178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:19:44.088096 13168 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 13:19:44.089052 13178 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:19:44.089237 13178 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92: Loaded TSK: 0
I20260925 13:19:44.089453 13178 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 13:19:44.185123 13170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:44.185424 13170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:44.185462 13170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:44.187194 13170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:44.187252 13170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:44.187294 13170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:19:44.189435 13170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35185
--tserver_master_addrs=127.0.98.62:44959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:44.189718 13170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:44.190001 13170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:44.196336 13208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:44.196347 13206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:44.196621 13205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:44.196859 13170 server_base.cc:1034] running on GCE node
I20260925 13:19:44.197083 13170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:44.197353 13170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:44.198530 13170 hybrid_clock.cc:648] HybridClock initialized: now 1790342384198511 us; error 34 us; skew 500 ppm
Sep 25 13:19:44 dist-test-slave-t8d9 krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:44.201694 13170 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:19:44.202229 13170 webserver.cc:460] Webserver started at http://127.0.98.1:35185/ using document root <none> and password file <none>
I20260925 13:19:44.202474 13170 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:44.202548 13170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 13:19:44 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:19:44.233914 13170 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.000s sys 0.001s
I20260925 13:19:44.234853 13216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:44.235060 13170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:19:44.235152 13170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "3304e037bbb1406f929346d0d599d0d3"
format_stamp: "Formatted at 2026-09-25 13:19:43 on dist-test-slave-t8d9"
server_key: "915db16e03a88765103cf9d400748a14"
server_key_iv: "3530aba94da0845263d03aa4eaa79f64"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:44.235625 13170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:44.251080 13170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:44.253404 13170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:44.253585 13170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:44.253850 13170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:44.254201 13170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:19:44.254252 13170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:44.254287 13170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:19:44.254303 13170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:44.261161 13170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46219
I20260925 13:19:44.261613 13329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46219 every 8 connection(s)
I20260925 13:19:44.261715 13170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:19:44.271205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 13170
I20260925 13:19:44.271507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46553
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44959
--webserver_port=45063
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43489/kms with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 13:19:44 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:44.285094 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:44.262627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44959 (local address 127.0.98.1:59673)
0925 13:19:44.262967 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:44.263024 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:44.263601 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:44.264154 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:44.264158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:44.264409 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:44.264685 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:44.264692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:44.266372 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:44.266373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:44.267193 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:44.267196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:44.268048 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:44.268049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:44.268152 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:44.272291 (+ 4139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:44.272302 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:44.273938 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:44.282843 (+ 8905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:44.282859 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:44.282861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:44.283243 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:44.283590 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:44.283591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:44.283591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:44.283771 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:44.284360 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:44.284368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:44.284558 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 13:19:44.284653 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 13:19:44.284837 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":89,"threads_started":1}
I20260925 13:19:44.285912 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:44.262678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59673 (local address 127.0.98.62:44959)
0925 13:19:44.262881 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:44.262890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:44.263164 (+ 274us) server_negotiation.cc:407] Connection header received
0925 13:19:44.263868 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:44.263877 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:44.263943 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:44.264096 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:44.264824 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:44.266265 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:44.267285 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:44.267934 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:44.268059 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:44.274073 (+ 6014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:44.274078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:44.274078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:44.274091 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:44.282669 (+ 8578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:44.283362 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:44.283366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:44.283366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:44.283494 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:44.283873 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:44.283874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:44.283875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:44.284253 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:44.285454 (+ 1201us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:44.285561 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 13:19:44.285704 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
I20260925 13:19:44.286626 13330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44959
I20260925 13:19:44.286734 13330 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:44.286940 13330 heartbeater.cc:511] Master 127.0.98.62:44959 requested a full tablet report, sending...
W20260925 13:19:44.287803 13330 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44959 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 13:19:44.287881 13330 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:44.287945 13330 heartbeater.cc:511] Master 127.0.98.62:44959 requested a full tablet report, sending...
I20260925 13:19:44.288264 13330 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:44.288343 13330 heartbeater.cc:511] Master 127.0.98.62:44959 requested a full tablet report, sending...
W20260925 13:19:44.288615 13330 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 13:19:44.375869 13334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:44.376158 13334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:44.376197 13334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:44.378046 13334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:44.378111 13334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:44.378153 13334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:19:44.380458 13334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45063
--tserver_master_addrs=127.0.98.62:44959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:44.380748 13334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:44.381011 13334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:44.386296 13342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:44.386310 13339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:44.386296 13340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:44.386996 13334 server_base.cc:1034] running on GCE node
I20260925 13:19:44.387192 13334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:44.387435 13334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:44.388577 13334 hybrid_clock.cc:648] HybridClock initialized: now 1790342384388567 us; error 26 us; skew 500 ppm
Sep 25 13:19:44 dist-test-slave-t8d9 krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:44.391148 13334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:19:44.391600 13334 webserver.cc:460] Webserver started at http://127.0.98.2:45063/ using document root <none> and password file <none>
I20260925 13:19:44.391810 13334 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:44.391872 13334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 13:19:44 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:19:44.414356 13334 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260925 13:19:44.415319 13350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:44.417768 13334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 13:19:44.418089 13334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6"
format_stamp: "Formatted at 2026-09-25 13:19:43 on dist-test-slave-t8d9"
server_key: "8c1ed468bf4acfdbbe454cedbc2b7c75"
server_key_iv: "68c1ca2f2fa5c60cad0ea691b7f92d52"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:44.418870 13334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:44.446952 13334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:44.449302 13334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:44.449440 13334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:44.449707 13334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:44.450049 13334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:19:44.450088 13334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:44.450117 13334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:19:44.450139 13334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:44.456212 13334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46553
I20260925 13:19:44.456355 13463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46553 every 8 connection(s)
I20260925 13:19:44.456630 13334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:19:44 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:44.466094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 13334
I20260925 13:19:44.466279 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:44.457343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33515 (local address 127.0.98.62:44959)
0925 13:19:44.457413 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:44.457415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:44.457595 (+ 180us) server_negotiation.cc:407] Connection header received
0925 13:19:44.458183 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:44.458184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:44.458200 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:44.458249 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:44.458804 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:44.459447 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:44.460415 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:44.460617 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:44.460673 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:44.463017 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:44.463019 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:44.463019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:44.463027 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:44.464585 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:44.465074 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:44.465075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:44.465075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:44.465136 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:44.465447 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:44.465448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:44.465448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:44.465633 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:44.465691 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:44.466098 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 13:19:44.466186 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 13:19:44.466378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40225
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44959
--webserver_port=41533
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43489/kms with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:19:44.466830 13466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:44.457296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44959 (local address 127.0.98.2:33515)
0925 13:19:44.457437 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:44.457487 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:44.458076 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:44.458322 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:44.458326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:44.458514 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:44.458709 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:44.458713 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:44.459538 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:44.459539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:44.460326 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:44.460328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:44.460700 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:44.460700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:44.460819 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:44.461355 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:44.461363 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:44.462908 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:44.464680 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:44.464687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:44.464690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:44.464978 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:44.465223 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:44.465224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:44.465224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:44.465367 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:44.465705 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:44.465712 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:44.465893 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 13:19:44.466422 (+ 529us) client_negotiation.cc:240] Negotiation successful
0925 13:19:44.466601 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260925 13:19:44.467269 13464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44959
I20260925 13:19:44.467360 13464 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:44.467549 13464 heartbeater.cc:511] Master 127.0.98.62:44959 requested a full tablet report, sending...
W20260925 13:19:44.468181 13464 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44959 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 13:19:44.468245 13464 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:44.468302 13464 heartbeater.cc:511] Master 127.0.98.62:44959 requested a full tablet report, sending...
I20260925 13:19:44.468595 13464 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:44.468662 13464 heartbeater.cc:511] Master 127.0.98.62:44959 requested a full tablet report, sending...
W20260925 13:19:44.468914 13464 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 13:19:44.579455 13467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:44.579672 13467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:44.579697 13467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:44.581511 13467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:44.581600 13467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:44.581640 13467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:19:44.583796 13467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41533
--tserver_master_addrs=127.0.98.62:44959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:44.584044 13467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:44.584275 13467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:44.591094 13474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:44.591252 13476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:44.591384 13467 server_base.cc:1034] running on GCE node
W20260925 13:19:44.592185 13473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:44.592392 13467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:44.592633 13467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:44.593782 13467 hybrid_clock.cc:648] HybridClock initialized: now 1790342384593772 us; error 27 us; skew 500 ppm
Sep 25 13:19:44 dist-test-slave-t8d9 krb5kdc[12136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:44.596400 13467 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:19:44.596865 13467 webserver.cc:460] Webserver started at http://127.0.98.3:41533/ using document root <none> and password file <none>
I20260925 13:19:44.597083 13467 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:44.597146 13467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 13:19:44 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:19:44.610728 13467 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:19:44.611423 13484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:44.611603 13467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:19:44.611690 13467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "add6026d4645419c984394e87260f3c4"
format_stamp: "Formatted at 2026-09-25 13:19:43 on dist-test-slave-t8d9"
server_key: "a90489a6531a9ea76b03f5e7b4fddce5"
server_key_iv: "429c6a071a087792908018f314c6c627"
server_key_version: "kuduclusterkey@0"
I20260925 13:19:44.612042 13467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:44.632385 13467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:44.634783 13467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:44.634931 13467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:44.635175 13467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:44.635545 13467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:19:44.635596 13467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:44.635634 13467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:19:44.635658 13467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:44.642858 13467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40225
I20260925 13:19:44.643086 13597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40225 every 8 connection(s)
I20260925 13:19:44.643545 13467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:19:44.643760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 13467
Sep 25 13:19:44 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:44.653788 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:44.644219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47700 (local address 127.0.98.62:44959)
0925 13:19:44.644270 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:44.644272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:44.644435 (+ 163us) server_negotiation.cc:407] Connection header received
0925 13:19:44.644529 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:44.644531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:44.644543 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:44.644597 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:44.645724 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:44.646495 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:44.647571 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:44.647850 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:44.647916 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:44.650666 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:44.650667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:44.650668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:44.650676 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:44.652406 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:44.652832 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:44.652833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:44.652833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:44.652896 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:44.653091 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:44.653091 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:44.653092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:44.653241 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:44.653411 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:44.653608 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 13:19:44.653684 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:19:44.655267 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:44.644688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44237 (local address 127.0.98.62:44959)
0925 13:19:44.645444 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:44.645446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:44.645453 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:19:44.645475 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:44.645476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:44.645487 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:44.645568 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:44.646213 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:44.646803 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:44.648235 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:44.648457 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:44.648519 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:44.651399 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:44.651401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:44.651401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:44.651409 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:44.653012 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:44.654188 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:44.654189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:44.654190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:44.654255 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:44.654585 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:44.654585 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:44.654586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:44.654738 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:44.654792 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:44.655099 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 13:19:44.655181 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":55,"threads_started":1}
I20260925 13:19:44.655442 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:44.644388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44959 (local address 127.0.98.3:44237)
0925 13:19:44.644585 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:44.644631 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:44.645258 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:44.645708 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:44.645713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:44.645908 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:44.646109 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:44.646114 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:44.646923 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:44.646924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:44.648130 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:44.648133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:44.648551 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:44.648552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:44.648679 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:44.649161 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:44.649172 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:44.651278 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:44.653447 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:44.653455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:44.653458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:44.654128 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:44.654354 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:44.654355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:44.654356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:44.654506 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:44.654794 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:44.654801 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:44.654961 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 13:19:44.655045 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 13:19:44.655207 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":97,"threads_started":1}
I20260925 13:19:44.655879 13598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44959
I20260925 13:19:44.655967 13598 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:44.656139 13598 heartbeater.cc:511] Master 127.0.98.62:44959 requested a full tablet report, sending...
W20260925 13:19:44.656734 13598 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44959 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 13:19:44.656798 13598 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:44.656862 13598 heartbeater.cc:511] Master 127.0.98.62:44959 requested a full tablet report, sending...
I20260925 13:19:44.657136 13598 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:44.657226 13598 heartbeater.cc:511] Master 127.0.98.62:44959 requested a full tablet report, sending...
W20260925 13:19:44.657498 13598 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 13:19:45 dist-test-slave-t8d9 krb5kdc[12136](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790342385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:45.288813 13330 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:45.288975 13330 heartbeater.cc:511] Master 127.0.98.62:44959 requested a full tablet report, sending...
I20260925 13:19:45.289430 13132 ts_manager.cc:194] Registered new tserver with Master: 3304e037bbb1406f929346d0d599d0d3 (127.0.98.1:46219)
I20260925 13:19:45.290524 13132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59673
I20260925 13:19:45.469100 13464 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:45.469269 13464 heartbeater.cc:511] Master 127.0.98.62:44959 requested a full tablet report, sending...
I20260925 13:19:45.469620 13132 ts_manager.cc:194] Registered new tserver with Master: e6dbca4bf9664fb2a33e47cd652ce7b6 (127.0.98.2:46553)
I20260925 13:19:45.470161 13132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33515
Sep 25 13:19:45 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 13:19:45.657724 13598 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:45.657876 13598 heartbeater.cc:511] Master 127.0.98.62:44959 requested a full tablet report, sending...
I20260925 13:19:45.658198 13132 ts_manager.cc:194] Registered new tserver with Master: add6026d4645419c984394e87260f3c4 (127.0.98.3:40225)
I20260925 13:19:45.658787 13132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44237
I20260925 13:19:45.666765 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:19:45.689288 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:45.669866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47710 (local address 127.0.98.62:44959)
0925 13:19:45.673673 (+ 3807us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:45.673675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:45.681745 (+ 8070us) server_negotiation.cc:407] Connection header received
0925 13:19:45.681798 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:45.681799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:45.681814 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:45.681876 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:45.682576 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.683379 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.684377 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.684687 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.684765 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:45.686291 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:45.686293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:45.686293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:45.686301 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:45.688091 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:45.688472 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:45.688473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:45.688473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:45.688542 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:45.688744 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:45.688745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:45.688746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:45.688891 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:45.688997 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:45.689098 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 13:19:45.689184 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3779,"thread_start_us":69,"threads_started":1}
I20260925 13:19:45.691620 13132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:19:45.694074 13132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:19:45.701891 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:45.697159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.0.98.2:46553)
0925 13:19:45.697476 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:45.697482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:45.697739 (+ 257us) server_negotiation.cc:407] Connection header received
0925 13:19:45.697797 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:45.697800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:45.697815 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:45.697877 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:45.698116 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.699204 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.700694 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.701448 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.701523 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:45.701643 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:45.701773 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 13:19:45.701814 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20260925 13:19:45.701900 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:45.697058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46553 (local address 127.0.0.1:36500)
0925 13:19:45.697552 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:45.697594 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:45.697661 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:45.697960 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:45.697962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:45.697964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:45.698045 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:45.698048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.699291 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.699292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:45.700607 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:45.700610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.701569 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.701570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:45.701656 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:45.701672 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 13:19:45.701754 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 13:19:45.701813 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":78,"threads_started":1}
I20260925 13:19:45.702879 13398 tablet_service.cc:1467] Processing CreateTablet for tablet ce40bbc66e004cc09b1997da2e36361f (DEFAULT_TABLE table=test-table [id=7889bd85d46c4073b68586ad7eaf1974]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:19:45.703209 13398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce40bbc66e004cc09b1997da2e36361f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:45.710628 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:45.697409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46219 (local address 127.0.0.1:52252)
0925 13:19:45.705608 (+ 8199us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:45.705620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:45.705681 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:45.705943 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:45.705944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:45.705945 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:45.705978 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:45.705979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.708365 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.708366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:45.709362 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:45.709365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.710396 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.710396 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:45.710457 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:45.710460 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:19:45.710499 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 13:19:45.710541 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8177,"thread_start_us":63,"threads_started":1}
I20260925 13:19:45.710933 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:45.697545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.0.98.1:46219)
0925 13:19:45.697656 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:45.697660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:45.705746 (+ 8086us) server_negotiation.cc:407] Connection header received
0925 13:19:45.705787 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:45.705789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:45.705804 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:45.705864 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:45.706047 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.707133 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.709462 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.710209 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.710261 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:45.710338 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:45.710809 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 13:19:45.710857 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":50,"threads_started":1}
I20260925 13:19:45.711685 13255 tablet_service.cc:1467] Processing CreateTablet for tablet ce40bbc66e004cc09b1997da2e36361f (DEFAULT_TABLE table=test-table [id=7889bd85d46c4073b68586ad7eaf1974]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:19:45.712061 13255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce40bbc66e004cc09b1997da2e36361f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:45.712880 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:45.697275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40225 (local address 127.0.0.1:35430)
0925 13:19:45.702075 (+ 4800us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:45.702085 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:45.702117 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:45.705880 (+ 3763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:45.705882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:45.705883 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:45.705918 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:45.705920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.707235 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.707237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:45.708269 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:45.708272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.712647 (+ 4375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.712648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:45.712711 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:45.712715 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:19:45.712765 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 13:19:45.712810 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4789,"thread_start_us":57,"threads_started":1}
I20260925 13:19:45.713294 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:45.697358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.0.98.3:40225)
0925 13:19:45.705650 (+ 8292us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:45.705656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:45.705667 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:19:45.705701 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:45.705703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:45.705719 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:45.705790 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:45.706045 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.707148 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.708375 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.712449 (+ 4074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.712523 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:45.712609 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:45.713168 (+ 559us) server_negotiation.cc:299] Negotiation successful
0925 13:19:45.713218 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8249,"thread_start_us":53,"threads_started":1}
I20260925 13:19:45.713976 13532 tablet_service.cc:1467] Processing CreateTablet for tablet ce40bbc66e004cc09b1997da2e36361f (DEFAULT_TABLE table=test-table [id=7889bd85d46c4073b68586ad7eaf1974]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:19:45.714346 13532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce40bbc66e004cc09b1997da2e36361f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:45.719599 13631 tablet_bootstrap.cc:492] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6: Bootstrap starting.
I20260925 13:19:45.719933 13632 tablet_bootstrap.cc:492] T ce40bbc66e004cc09b1997da2e36361f P 3304e037bbb1406f929346d0d599d0d3: Bootstrap starting.
I20260925 13:19:45.720157 13633 tablet_bootstrap.cc:492] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4: Bootstrap starting.
I20260925 13:19:45.720314 13631 tablet_bootstrap.cc:654] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:45.720633 13631 log.cc:824] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:45.721252 13633 tablet_bootstrap.cc:654] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:45.721524 13632 tablet_bootstrap.cc:654] T ce40bbc66e004cc09b1997da2e36361f P 3304e037bbb1406f929346d0d599d0d3: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:45.721819 13632 log.cc:824] T ce40bbc66e004cc09b1997da2e36361f P 3304e037bbb1406f929346d0d599d0d3: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:45.722185 13633 log.cc:824] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:45.722749 13631 tablet_bootstrap.cc:492] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6: No bootstrap required, opened a new log
I20260925 13:19:45.722823 13631 ts_tablet_manager.cc:1397] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 13:19:45.722903 13633 tablet_bootstrap.cc:492] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4: No bootstrap required, opened a new log
I20260925 13:19:45.722958 13633 ts_tablet_manager.cc:1397] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 13:19:45.725464 13631 raft_consensus.cc:348] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46553 } } peers { permanent_uuid: "add6026d4645419c984394e87260f3c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } } peers { permanent_uuid: "3304e037bbb1406f929346d0d599d0d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46219 } }
I20260925 13:19:45.725660 13631 raft_consensus.cc:374] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:45.725528 13633 raft_consensus.cc:348] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46553 } } peers { permanent_uuid: "add6026d4645419c984394e87260f3c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } } peers { permanent_uuid: "3304e037bbb1406f929346d0d599d0d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46219 } }
I20260925 13:19:45.725705 13631 raft_consensus.cc:729] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dbca4bf9664fb2a33e47cd652ce7b6, State: Initialized, Role: FOLLOWER
I20260925 13:19:45.725718 13633 raft_consensus.cc:374] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:45.725750 13633 raft_consensus.cc:729] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add6026d4645419c984394e87260f3c4, State: Initialized, Role: FOLLOWER
I20260925 13:19:45.725883 13633 consensus_queue.cc:260] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46553 } } peers { permanent_uuid: "add6026d4645419c984394e87260f3c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } } peers { permanent_uuid: "3304e037bbb1406f929346d0d599d0d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46219 } }
I20260925 13:19:45.726152 13633 ts_tablet_manager.cc:1428] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 13:19:45.726114 13631 consensus_queue.cc:260] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46553 } } peers { permanent_uuid: "add6026d4645419c984394e87260f3c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } } peers { permanent_uuid: "3304e037bbb1406f929346d0d599d0d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46219 } }
I20260925 13:19:45.726282 13598 heartbeater.cc:503] Master 127.0.98.62:44959 was elected leader, sending a full tablet report...
I20260925 13:19:45.726450 13631 ts_tablet_manager.cc:1428] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 13:19:45.726580 13464 heartbeater.cc:503] Master 127.0.98.62:44959 was elected leader, sending a full tablet report...
I20260925 13:19:45.728403 13632 tablet_bootstrap.cc:492] T ce40bbc66e004cc09b1997da2e36361f P 3304e037bbb1406f929346d0d599d0d3: No bootstrap required, opened a new log
I20260925 13:19:45.728484 13632 ts_tablet_manager.cc:1397] T ce40bbc66e004cc09b1997da2e36361f P 3304e037bbb1406f929346d0d599d0d3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260925 13:19:45.731426 13632 raft_consensus.cc:348] T ce40bbc66e004cc09b1997da2e36361f P 3304e037bbb1406f929346d0d599d0d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46553 } } peers { permanent_uuid: "add6026d4645419c984394e87260f3c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } } peers { permanent_uuid: "3304e037bbb1406f929346d0d599d0d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46219 } }
I20260925 13:19:45.731606 13632 raft_consensus.cc:374] T ce40bbc66e004cc09b1997da2e36361f P 3304e037bbb1406f929346d0d599d0d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:45.731637 13632 raft_consensus.cc:729] T ce40bbc66e004cc09b1997da2e36361f P 3304e037bbb1406f929346d0d599d0d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3304e037bbb1406f929346d0d599d0d3, State: Initialized, Role: FOLLOWER
I20260925 13:19:45.731789 13632 consensus_queue.cc:260] T ce40bbc66e004cc09b1997da2e36361f P 3304e037bbb1406f929346d0d599d0d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46553 } } peers { permanent_uuid: "add6026d4645419c984394e87260f3c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } } peers { permanent_uuid: "3304e037bbb1406f929346d0d599d0d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46219 } }
I20260925 13:19:45.732040 13632 ts_tablet_manager.cc:1428] T ce40bbc66e004cc09b1997da2e36361f P 3304e037bbb1406f929346d0d599d0d3: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260925 13:19:45.732091 13330 heartbeater.cc:503] Master 127.0.98.62:44959 was elected leader, sending a full tablet report...
W20260925 13:19:45.763283 13331 tablet.cc:2369] T ce40bbc66e004cc09b1997da2e36361f P 3304e037bbb1406f929346d0d599d0d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 13:19:45 dist-test-slave-t8d9 krb5kdc[12136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260925 13:19:45.894961 13599 tablet.cc:2369] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:19:45.957919 13465 tablet.cc:2369] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:19:45.975376 13638 raft_consensus.cc:482] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:19:45.975535 13638 raft_consensus.cc:504] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46553 } } peers { permanent_uuid: "add6026d4645419c984394e87260f3c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } } peers { permanent_uuid: "3304e037bbb1406f929346d0d599d0d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46219 } }
I20260925 13:19:45.975932 13638 leader_election.cc:288] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers add6026d4645419c984394e87260f3c4 (127.0.98.3:40225), 3304e037bbb1406f929346d0d599d0d3 (127.0.98.1:46219)
I20260925 13:19:45.981364 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:45.976837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46821 (local address 127.0.98.1:46219)
0925 13:19:45.976891 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:45.976893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:45.977462 (+ 569us) server_negotiation.cc:407] Connection header received
0925 13:19:45.977576 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:45.977577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:45.977590 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:45.977621 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:45.978370 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.979105 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.980361 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.980956 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.981017 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:45.981029 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:45.981222 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 13:19:45.981262 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 13:19:45.982198 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:45.976458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46219 (local address 127.0.98.2:46821)
0925 13:19:45.977412 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:45.977420 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:45.977484 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:45.977687 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:45.977688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:45.977690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:45.978262 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:45.978266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.979197 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.979197 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:45.980258 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:45.980261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.981086 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.981086 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:45.981152 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:45.981155 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:19:45.982042 (+ 887us) client_negotiation.cc:240] Negotiation successful
0925 13:19:45.982092 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"thread_start_us":65,"threads_started":1}
I20260925 13:19:45.982683 13277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce40bbc66e004cc09b1997da2e36361f" candidate_uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3304e037bbb1406f929346d0d599d0d3" is_pre_election: true
I20260925 13:19:45.982862 13277 raft_consensus.cc:2454] T ce40bbc66e004cc09b1997da2e36361f P 3304e037bbb1406f929346d0d599d0d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dbca4bf9664fb2a33e47cd652ce7b6 in term 0.
I20260925 13:19:45.983224 13352 leader_election.cc:302] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3304e037bbb1406f929346d0d599d0d3, e6dbca4bf9664fb2a33e47cd652ce7b6; no voters:
I20260925 13:19:45.983364 13638 raft_consensus.cc:2781] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:19:45.983410 13638 raft_consensus.cc:482] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:19:45.983433 13638 raft_consensus.cc:3037] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:45.984386 13638 raft_consensus.cc:504] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46553 } } peers { permanent_uuid: "add6026d4645419c984394e87260f3c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } } peers { permanent_uuid: "3304e037bbb1406f929346d0d599d0d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46219 } }
I20260925 13:19:45.984519 13638 leader_election.cc:288] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [CANDIDATE]: Term 1 election: Requested vote from peers add6026d4645419c984394e87260f3c4 (127.0.98.3:40225), 3304e037bbb1406f929346d0d599d0d3 (127.0.98.1:46219)
I20260925 13:19:45.985010 13277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce40bbc66e004cc09b1997da2e36361f" candidate_uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3304e037bbb1406f929346d0d599d0d3"
I20260925 13:19:45.985090 13277 raft_consensus.cc:3037] T ce40bbc66e004cc09b1997da2e36361f P 3304e037bbb1406f929346d0d599d0d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:45.986081 13277 raft_consensus.cc:2454] T ce40bbc66e004cc09b1997da2e36361f P 3304e037bbb1406f929346d0d599d0d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dbca4bf9664fb2a33e47cd652ce7b6 in term 1.
I20260925 13:19:45.986326 13352 leader_election.cc:302] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3304e037bbb1406f929346d0d599d0d3, e6dbca4bf9664fb2a33e47cd652ce7b6; no voters:
I20260925 13:19:45.986450 13638 raft_consensus.cc:2781] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:19:45.986603 13638 raft_consensus.cc:686] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [term 1 LEADER]: Becoming Leader. State: Replica: e6dbca4bf9664fb2a33e47cd652ce7b6, State: Running, Role: LEADER
I20260925 13:19:45.986696 13638 consensus_queue.cc:237] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46553 } } peers { permanent_uuid: "add6026d4645419c984394e87260f3c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } } peers { permanent_uuid: "3304e037bbb1406f929346d0d599d0d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46219 } }
I20260925 13:19:45.987739 13131 catalog_manager.cc:5495] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6dbca4bf9664fb2a33e47cd652ce7b6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "add6026d4645419c984394e87260f3c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3304e037bbb1406f929346d0d599d0d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46219 } health_report { overall_health: UNKNOWN } } }
I20260925 13:19:45.990697 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:45.976249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40225 (local address 127.0.98.2:47867)
0925 13:19:45.977234 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:45.977248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:45.977328 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:45.977564 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:45.977566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:45.977568 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:45.977667 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:45.977669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.978523 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.978524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:45.979754 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:45.979757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.990445 (+ 10688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.990446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:45.990523 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:45.990527 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:19:45.990571 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 13:19:45.990618 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"thread_start_us":104,"threads_started":1}
I20260925 13:19:45.991146 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:45.976782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47867 (local address 127.0.98.3:40225)
0925 13:19:45.976842 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:45.976845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:45.977314 (+ 469us) server_negotiation.cc:407] Connection header received
0925 13:19:45.977386 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:45.977388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:45.977402 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:45.977487 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:45.977749 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.978409 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.989645 (+ 11236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:45.990306 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:45.990382 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:45.990394 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:45.991017 (+ 623us) server_negotiation.cc:299] Negotiation successful
0925 13:19:45.991076 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 13:19:45.991449 13552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce40bbc66e004cc09b1997da2e36361f" candidate_uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add6026d4645419c984394e87260f3c4" is_pre_election: true
I20260925 13:19:45.991591 13552 raft_consensus.cc:2454] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dbca4bf9664fb2a33e47cd652ce7b6 in term 0.
I20260925 13:19:45.991709 13551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce40bbc66e004cc09b1997da2e36361f" candidate_uuid: "e6dbca4bf9664fb2a33e47cd652ce7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add6026d4645419c984394e87260f3c4"
I20260925 13:19:45.991760 13551 raft_consensus.cc:3037] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:45.992666 13551 raft_consensus.cc:2454] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dbca4bf9664fb2a33e47cd652ce7b6 in term 1.
I20260925 13:19:46.056816 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:46.053991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.0.98.2:46553)
0925 13:19:46.054033 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:46.054035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:46.054077 (+ 42us) server_negotiation.cc:407] Connection header received
0925 13:19:46.054171 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:46.054171 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:46.054192 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:46.054249 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:19:46.054403 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:46.054992 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:46.056060 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:46.056259 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:46.056314 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:46.056484 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:19:46.056592 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:19:46.056647 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:46.056724 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 13:19:46.056745 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:19:46.058758 13277 raft_consensus.cc:1261] T ce40bbc66e004cc09b1997da2e36361f P 3304e037bbb1406f929346d0d599d0d3 [term 1 FOLLOWER]: Refusing update from remote peer e6dbca4bf9664fb2a33e47cd652ce7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:19:46.058758 13551 raft_consensus.cc:1261] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4 [term 1 FOLLOWER]: Refusing update from remote peer e6dbca4bf9664fb2a33e47cd652ce7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:19:46.059175 13644 consensus_queue.cc:1035] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3304e037bbb1406f929346d0d599d0d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:19:46.059329 13638 consensus_queue.cc:1035] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "add6026d4645419c984394e87260f3c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:19:46.061636 13654 mvcc.cc:205] Tried to move back new op lower bound from 7333242413294067712 to 7333242413002170368. Current Snapshot: MvccSnapshot[applied={T|T < 7333242413294067712}]
I20260925 13:19:46.063102 13655 mvcc.cc:205] Tried to move back new op lower bound from 7333242413294067712 to 7333242413002170368. Current Snapshot: MvccSnapshot[applied={T|T < 7333242413294067712 or (T in {7333242413294067712})}]
I20260925 13:19:46.065295 13132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:19:46.065390 13132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:19:46.066124 13132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bc95a2806a04e1b8157a65e295aec92: Sending DeleteTablet for 3 replicas of tablet ce40bbc66e004cc09b1997da2e36361f
I20260925 13:19:46.066536 13255 tablet_service.cc:1514] Processing DeleteTablet for tablet ce40bbc66e004cc09b1997da2e36361f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:19:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52252
I20260925 13:19:46.066592 13532 tablet_service.cc:1514] Processing DeleteTablet for tablet ce40bbc66e004cc09b1997da2e36361f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:19:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35430
I20260925 13:19:46.067034 13398 tablet_service.cc:1514] Processing DeleteTablet for tablet ce40bbc66e004cc09b1997da2e36361f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:19:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36500
I20260925 13:19:46.067111 13664 tablet_replica.cc:330] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4: stopping tablet replica
I20260925 13:19:46.067219 13664 raft_consensus.cc:2229] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:19:46.067282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 13170
I20260925 13:19:46.067327 13664 raft_consensus.cc:2258] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:19:46.069205 13664 ts_tablet_manager.cc:1905] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:19:46.071175 13664 ts_tablet_manager.cc:1918] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:19:46.071309 13664 log.cc:1196] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/ce40bbc66e004cc09b1997da2e36361f
I20260925 13:19:46.071659 13664 ts_tablet_manager.cc:1939] T ce40bbc66e004cc09b1997da2e36361f P add6026d4645419c984394e87260f3c4: Deleting consensus metadata
I20260925 13:19:46.072184 13117 catalog_manager.cc:4841] TS add6026d4645419c984394e87260f3c4 (127.0.98.3:40225): tablet ce40bbc66e004cc09b1997da2e36361f (table test-table [id=7889bd85d46c4073b68586ad7eaf1974]) successfully deleted
I20260925 13:19:46.073714 13665 tablet_replica.cc:330] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6: stopping tablet replica
I20260925 13:19:46.073901 13665 raft_consensus.cc:2229] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:19:46.074091 13665 raft_consensus.cc:2258] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:19:46.074541 13665 ts_tablet_manager.cc:1905] T ce40bbc66e004cc09b1997da2e36361f P e6dbca4bf9664fb2a33e47cd652ce7b6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 13:19:46.074718 13117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46219 (error 108)
I20260925 13:19:46.075511 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:46.075391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46219 (local address 127.0.0.1:52258)
0925 13:19:46.075441 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:46.075475 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 13:19:46.075630 13117 catalog_manager.cc:4584] TS 3304e037bbb1406f929346d0d599d0d3 (127.0.98.1:46219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce40bbc66e004cc09b1997da2e36361f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46219: connect: Connection refused (error 111)
I20260925 13:19:46.075839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 13334
I20260925 13:19:46.080176 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 13467
I20260925 13:19:46.080256 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:46.080129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46553 (local address 127.0.0.1:36518)
0925 13:19:46.080194 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:46.080221 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 13:19:46.084507 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 13099
I20260925 13:19:46.089805 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 13:19:46.423900 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 13:19:46.423974 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 13:19:46.758136 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 13:19:46.758 UTC [12166] LOG: received smart shutdown request
2026-09-25 13:19:46.761 UTC [12166] LOG: background worker "logical replication launcher" (PID 12175) exited with exit code 1
2026-09-25 13:19:46.761 UTC [12170] LOG: shutting down
2026-09-25 13:19:46.927 UTC [12166] LOG: database system is shut down
2026-09-25T13:19:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45340 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:19:47 dist-test-slave-t8d9 krb5kdc[13675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:19:47 dist-test-slave-t8d9 krb5kdc[13675](info): set up 2 sockets
Sep 25 13:19:47 dist-test-slave-t8d9 krb5kdc[13675](info): commencing operation
krb5kdc: starting...
W20260925 13:19:47.969655 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:19:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:19:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:19:48.081070 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37213
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37213,127.0.98.61:36019,127.0.98.60:37673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:19:48.159215 13688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:48.159427 13688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:48.159449 13688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:48.160600 13688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:19:48.160642 13688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:48.160655 13688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:48.160666 13688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:19:48.160677 13688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:19:48.162230 13688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37213,127.0.98.61:36019,127.0.98.60:37673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:48.162456 13688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:48.162683 13688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:48.167336 13696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:48.167344 13693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:48.167348 13694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:48.167652 13688 server_base.cc:1034] running on GCE node
I20260925 13:19:48.167877 13688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:48.168133 13688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:48.169267 13688 hybrid_clock.cc:648] HybridClock initialized: now 1790342388169254 us; error 29 us; skew 500 ppm
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:48.171622 13688 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:19:48.171998 13688 webserver.cc:460] Webserver started at http://127.0.98.62:34687/ using document root <none> and password file <none>
I20260925 13:19:48.172186 13688 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:48.172225 13688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:48.172314 13688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:19:48.173290 13688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "869bb6b5f6984488b69f27549670e8e6"
format_stamp: "Formatted at 2026-09-25 13:19:48 on dist-test-slave-t8d9"
server_key: "092ca02cbf5327c55b96314b10d8ae56"
server_key_iv: "a242ad6ec054919320c0e386b183812e"
server_key_version: "encryptionkey@0"
I20260925 13:19:48.173625 13688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "869bb6b5f6984488b69f27549670e8e6"
format_stamp: "Formatted at 2026-09-25 13:19:48 on dist-test-slave-t8d9"
server_key: "092ca02cbf5327c55b96314b10d8ae56"
server_key_iv: "a242ad6ec054919320c0e386b183812e"
server_key_version: "encryptionkey@0"
I20260925 13:19:48.174772 13688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:19:48.175444 13703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:48.175590 13688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:19:48.175670 13688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "869bb6b5f6984488b69f27549670e8e6"
format_stamp: "Formatted at 2026-09-25 13:19:48 on dist-test-slave-t8d9"
server_key: "092ca02cbf5327c55b96314b10d8ae56"
server_key_iv: "a242ad6ec054919320c0e386b183812e"
server_key_version: "encryptionkey@0"
I20260925 13:19:48.175735 13688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:48.188871 13688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:48.191067 13688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:48.191190 13688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:48.194890 13688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37213
I20260925 13:19:48.194926 13755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37213 every 8 connection(s)
I20260925 13:19:48.195276 13688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:19:48.195443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 13688
I20260925 13:19:48.195528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:19:48.195729 392 external_mini_cluster.cc:1428] Setting key 23068a0695790def71bc1b613af2847c
I20260925 13:19:48.196132 13756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:48.197693 13756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:48.210278 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.198426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37213 (local address 127.0.0.1:34214)
0925 13:19:48.198622 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.198666 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:48.199202 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:48.199545 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:48.199547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:48.199844 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:48.200081 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.200090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.201093 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.201095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.202203 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.202206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.202684 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.202685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.202769 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.203309 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:48.203319 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:48.207204 (+ 3885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:48.208856 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.208862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.208864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.209043 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.209276 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.209276 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.209277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.209351 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.209729 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:48.209734 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:48.209798 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 13:19:48.209879 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 13:19:48.210098 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":81,"thread_start_us":92,"threads_started":1}
I20260925 13:19:48.210280 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.198645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.0.98.62:37213)
0925 13:19:48.198803 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:48.198808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:48.198849 (+ 41us) server_negotiation.cc:407] Connection header received
0925 13:19:48.199371 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:48.199374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:48.199399 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:48.199506 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:48.200187 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.200994 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.202288 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.202593 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.202668 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.207297 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:48.207307 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:48.207307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:48.207314 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:48.208780 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.209130 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.209133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.209133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.209214 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.209407 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.209407 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.209408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.209669 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:48.209745 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:48.209911 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 13:19:48.210028 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260925 13:19:48.211103 13756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:19:48.211429 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.211321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36019 (local address 127.0.0.1:42984)
0925 13:19:48.211363 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.211405 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 13:19:48.211553 13707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36019: connect: Connection refused (error 111)
I20260925 13:19:48.211925 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.211841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36019 (local address 127.0.0.1:42992)
0925 13:19:48.211881 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.211904 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 13:19:48.212047 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36019: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36019: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 13:19:48.221823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36019
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37213,127.0.98.61:36019,127.0.98.60:37673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:19:48.261260 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } attempt: 1
I20260925 13:19:48.261816 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.261683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36019 (local address 127.0.0.1:43000)
0925 13:19:48.261744 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.261786 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 13:19:48.262156 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.262061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36019 (local address 127.0.0.1:43014)
0925 13:19:48.262106 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.262133 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 13:19:48.262276 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36019: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36019: connect: Connection refused (error 111)
W20260925 13:19:48.301725 13763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:48.301927 13763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:48.301957 13763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:48.303078 13763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:19:48.303124 13763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:48.303143 13763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:48.303158 13763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:19:48.303172 13763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:19:48.304668 13763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37213,127.0.98.61:36019,127.0.98.60:37673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:48.304908 13763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:48.305124 13763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:48.309779 13771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:48.309773 13769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:48.309734 13768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:48.309922 13763 server_base.cc:1034] running on GCE node
I20260925 13:19:48.310199 13763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:48.310456 13763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:48.311589 13763 hybrid_clock.cc:648] HybridClock initialized: now 1790342388311575 us; error 28 us; skew 500 ppm
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:48.313935 13763 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 13:19:48.314361 13763 webserver.cc:460] Webserver started at http://127.0.98.61:35743/ using document root <none> and password file <none>
I20260925 13:19:48.314566 13763 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:48.314620 13763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:48.314742 13763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:19:48.315760 13763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/data/instance:
uuid: "b4a5a9c40e654d94a5bad833ca6f1566"
format_stamp: "Formatted at 2026-09-25 13:19:48 on dist-test-slave-t8d9"
server_key: "e2602ff1993bcbe83518003694664fa1"
server_key_iv: "6dc3d98ca21fea9402b35c554c4fe531"
server_key_version: "encryptionkey@0"
I20260925 13:19:48.316098 13763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/wal/instance:
uuid: "b4a5a9c40e654d94a5bad833ca6f1566"
format_stamp: "Formatted at 2026-09-25 13:19:48 on dist-test-slave-t8d9"
server_key: "e2602ff1993bcbe83518003694664fa1"
server_key_iv: "6dc3d98ca21fea9402b35c554c4fe531"
server_key_version: "encryptionkey@0"
I20260925 13:19:48.317266 13763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:19:48.317957 13778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:48.318107 13763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:48.318185 13763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/wal
uuid: "b4a5a9c40e654d94a5bad833ca6f1566"
format_stamp: "Formatted at 2026-09-25 13:19:48 on dist-test-slave-t8d9"
server_key: "e2602ff1993bcbe83518003694664fa1"
server_key_iv: "6dc3d98ca21fea9402b35c554c4fe531"
server_key_version: "encryptionkey@0"
I20260925 13:19:48.318250 13763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:48.326445 13763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:48.328140 13763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:48.328275 13763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:48.330394 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } attempt: 2
I20260925 13:19:48.330791 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.330647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36019 (local address 127.0.0.1:43026)
0925 13:19:48.330709 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.330753 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 13:19:48.331126 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.331016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36019 (local address 127.0.0.1:43034)
0925 13:19:48.331065 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.331103 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 13:19:48.331221 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36019: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36019: connect: Connection refused (error 111)
I20260925 13:19:48.332485 13763 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36019
I20260925 13:19:48.332511 13830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36019 every 8 connection(s)
I20260925 13:19:48.332974 13763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/data/info.pb
I20260925 13:19:48.334002 13831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:48.335497 13831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:19:48.336297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 13763
I20260925 13:19:48.336390 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-1/wal/instance
I20260925 13:19:48.336566 392 external_mini_cluster.cc:1428] Setting key c84a05dbb311e1c21f322a1cbe4c658b
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260925 13:19:48.344463 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.336157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.0.98.62:37213)
0925 13:19:48.336214 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:48.336217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:48.336411 (+ 194us) server_negotiation.cc:407] Connection header received
0925 13:19:48.337023 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:48.337025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:48.337034 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:48.337072 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:48.337837 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.338309 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.339290 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.339569 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.339637 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.341907 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:48.341908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:48.341908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:48.341914 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:48.343149 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.343600 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.343600 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.343601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.343644 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.343880 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.343881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.343881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.343972 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:48.344031 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:48.344335 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 13:19:48.344397 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:19:48.344625 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.336131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37213 (local address 127.0.0.1:34220)
0925 13:19:48.336287 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.336333 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:48.336915 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:48.337146 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:48.337150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:48.337463 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:48.337765 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.337771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.338388 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.338389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.339204 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.339206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.339640 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.339640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.339748 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.340238 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:48.340248 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:48.341820 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:48.343237 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.343244 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.343246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.343528 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.343699 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.343700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.343700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.343827 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.344037 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:48.344043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:48.344210 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 13:19:48.344316 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 13:19:48.344435 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20260925 13:19:48.345152 13831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:19:48.351940 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.345382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36019 (local address 127.0.0.1:43050)
0925 13:19:48.345429 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.345439 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:48.345477 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:48.345685 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:48.345685 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:48.345812 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:48.345854 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.345856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.346788 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.346789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.347274 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.347275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.347780 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.347781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.347848 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.348197 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:48.348202 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:48.349345 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:48.351069 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.351070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.351071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.351224 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.351424 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.351424 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.351425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.351468 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.351727 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:48.351728 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:48.351759 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 13:19:48.351802 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:19:48.351875 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 13:19:48.352059 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.345413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.0.98.61:36019)
0925 13:19:48.345562 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:48.345566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:48.345585 (+ 19us) server_negotiation.cc:407] Connection header received
0925 13:19:48.345603 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:48.345605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:48.345615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:48.345673 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:48.345917 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.346713 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.347346 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.347713 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.347766 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.349428 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:48.349436 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:48.349436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:48.349444 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:48.350996 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.351301 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.351304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.351305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.351366 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.351502 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.351502 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.351503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.351669 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:48.351741 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:48.351837 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 13:19:48.351936 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260925 13:19:48.352620 13831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:19:48.352943 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.352842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37673 (local address 127.0.0.1:59954)
0925 13:19:48.352883 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.352924 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 13:19:48.353085 13779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
I20260925 13:19:48.353473 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.353393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37673 (local address 127.0.0.1:59958)
0925 13:19:48.353434 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.353457 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 13:19:48.353588 13831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37673: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 13:19:48.360929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37673
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37213,127.0.98.61:36019,127.0.98.60:37673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:19:48.402799 13831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } attempt: 1
I20260925 13:19:48.403270 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.403139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37673 (local address 127.0.0.1:59966)
0925 13:19:48.403204 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.403239 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 13:19:48.403604 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.403514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37673 (local address 127.0.0.1:59972)
0925 13:19:48.403559 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.403584 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 13:19:48.403714 13831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37673: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
I20260925 13:19:48.422363 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } attempt: 3
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:19:48.428874 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.422713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36019 (local address 127.0.0.1:43066)
0925 13:19:48.422769 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.422781 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:48.422831 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:48.422946 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:48.422947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:48.423100 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:48.423162 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.423163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.423862 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.423863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.424409 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.424412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.424774 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.424775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.424851 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.425341 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:48.425345 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:48.426560 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:48.427922 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.427923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.427923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.428087 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.428287 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.428288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.428288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.428346 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.428561 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:48.428562 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:48.428610 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 13:19:48.428668 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 13:19:48.428778 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 13:19:48.428874 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.422751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.0.98.61:36019)
0925 13:19:48.422787 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:48.422789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:48.422814 (+ 25us) server_negotiation.cc:407] Connection header received
0925 13:19:48.422882 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:48.422882 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:48.422894 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:48.422941 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:48.423237 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.423778 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.424486 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.424709 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.424765 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.426641 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:48.426643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:48.426643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:48.426651 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:48.427843 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.428149 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.428150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.428151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.428229 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.428397 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.428398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.428398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.428506 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:48.428558 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:48.428697 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 13:19:48.428779 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:19:48.429261 13756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:19:48.429595 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.429480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37673 (local address 127.0.0.1:59988)
0925 13:19:48.429523 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.429569 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 13:19:48.429934 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.429855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37673 (local address 127.0.0.1:60004)
0925 13:19:48.429896 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.429918 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 13:19:48.430022 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37673: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
W20260925 13:19:48.443481 13838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:48.443702 13838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:48.443725 13838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:48.444867 13838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:19:48.444906 13838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:48.444919 13838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:48.444931 13838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:19:48.444957 13838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:19:48.446543 13838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37213,127.0.98.61:36019,127.0.98.60:37673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:48.446759 13838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:48.446939 13838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:48.451438 13843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:48.451464 13844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:48.451658 13838 server_base.cc:1034] running on GCE node
W20260925 13:19:48.451438 13846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:48.451951 13838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:48.452236 13838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:48.453380 13838 hybrid_clock.cc:648] HybridClock initialized: now 1790342388453358 us; error 36 us; skew 500 ppm
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:48.455708 13838 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 13:19:48.456151 13838 webserver.cc:460] Webserver started at http://127.0.98.60:40031/ using document root <none> and password file <none>
I20260925 13:19:48.456369 13838 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:48.456416 13838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:48.456540 13838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:19:48.457587 13838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/data/instance:
uuid: "b49b14c6d3574f2489993c8d81fcb1f4"
format_stamp: "Formatted at 2026-09-25 13:19:48 on dist-test-slave-t8d9"
server_key: "66738cd3ef1ff0afa785309d7701f7cf"
server_key_iv: "6de704458f74532cbac554c455f038df"
server_key_version: "encryptionkey@0"
I20260925 13:19:48.457921 13838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/wal/instance:
uuid: "b49b14c6d3574f2489993c8d81fcb1f4"
format_stamp: "Formatted at 2026-09-25 13:19:48 on dist-test-slave-t8d9"
server_key: "66738cd3ef1ff0afa785309d7701f7cf"
server_key_iv: "6de704458f74532cbac554c455f038df"
server_key_version: "encryptionkey@0"
I20260925 13:19:48.459118 13838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:19:48.459758 13853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:48.459903 13838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:19:48.459981 13838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/wal
uuid: "b49b14c6d3574f2489993c8d81fcb1f4"
format_stamp: "Formatted at 2026-09-25 13:19:48 on dist-test-slave-t8d9"
server_key: "66738cd3ef1ff0afa785309d7701f7cf"
server_key_iv: "6de704458f74532cbac554c455f038df"
server_key_version: "encryptionkey@0"
I20260925 13:19:48.460048 13838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:48.461189 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } attempt: 1
I20260925 13:19:48.461603 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.461447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37673 (local address 127.0.0.1:60016)
0925 13:19:48.461502 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.461571 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 13:19:48.461881 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.461821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37673 (local address 127.0.0.1:60032)
0925 13:19:48.461844 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.461866 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260925 13:19:48.461963 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37673: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
I20260925 13:19:48.471874 13831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } attempt: 2
I20260925 13:19:48.472252 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.472139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37673 (local address 127.0.0.1:60034)
0925 13:19:48.472197 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.472224 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 13:19:48.472572 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.472471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37673 (local address 127.0.0.1:60040)
0925 13:19:48.472515 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.472548 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 13:19:48.472680 13831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37673: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37673: connect: Connection refused (error 111)
I20260925 13:19:48.475987 13838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:48.478030 13838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:48.478176 13838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:48.481936 13838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37673
I20260925 13:19:48.481963 13905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37673 every 8 connection(s)
I20260925 13:19:48.482280 13838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/data/info.pb
I20260925 13:19:48.483036 13906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:48.484514 13906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:19:48.485149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 13838
I20260925 13:19:48.485247 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790342239834770-392-0/minicluster-data/master-2/wal/instance
I20260925 13:19:48.485430 392 external_mini_cluster.cc:1428] Setting key 4c59a6f9c535da858daf1ab75d2bdde5
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:19:48.493472 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.485184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.0.98.62:37213)
0925 13:19:48.485252 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:48.485255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:48.485520 (+ 265us) server_negotiation.cc:407] Connection header received
0925 13:19:48.486134 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:48.486135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:48.486144 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:48.486185 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:48.486706 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.487185 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.488182 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.488469 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.488528 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.490813 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:48.490814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:48.490814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:48.490822 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:48.492207 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.492626 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.492627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.492627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.492681 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.492895 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.492895 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.492896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.493005 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:48.493057 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:48.493304 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 13:19:48.493381 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 13:19:48.493697 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.485823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.0.98.60:37673)
0925 13:19:48.485964 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:48.485968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:48.486000 (+ 32us) server_negotiation.cc:407] Connection header received
0925 13:19:48.486082 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:48.486090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:48.486114 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:48.486207 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:48.486753 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.487706 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.488500 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.488899 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.488957 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.491202 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:48.491209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:48.491210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:48.491216 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:48.492551 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.492852 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.492854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.492855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.492900 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.493060 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.493061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.493062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.493206 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:48.493335 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:48.493405 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 13:19:48.493478 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260925 13:19:48.493803 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.485131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37213 (local address 127.0.0.1:34222)
0925 13:19:48.485387 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.485433 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:48.486043 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:48.486211 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:48.486213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:48.486474 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:48.486619 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.486624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.487257 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.487258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.488109 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.488111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.488531 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.488532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.488673 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.489188 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:48.489198 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:48.490729 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:48.492282 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.492288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.492290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.492558 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.492734 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.492735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.492735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.492838 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.493061 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:48.493066 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:48.493196 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 13:19:48.493272 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 13:19:48.493412 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":66,"threads_started":1}
I20260925 13:19:48.494274 13906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:48.500077 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.494563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.0.98.61:36019)
0925 13:19:48.494607 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:48.494610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:48.494615 (+ 5us) server_negotiation.cc:407] Connection header received
0925 13:19:48.494634 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:48.494635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:48.494642 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:48.494684 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:48.494935 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.495385 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.495983 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.496174 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.496231 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.497759 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:48.497760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:48.497760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:48.497766 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:48.499177 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.499448 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.499449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.499450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.499504 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.499641 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.499641 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.499642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.499763 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:48.499823 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:48.499951 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 13:19:48.500008 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:19:48.500092 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.494490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36019 (local address 127.0.0.1:43076)
0925 13:19:48.494526 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.494534 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:48.494564 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:48.494708 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:48.494709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:48.494826 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:48.494869 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.494870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.495450 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.495450 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.495916 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.495917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.496235 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.496236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.496299 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.496588 (+ 289us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:48.496592 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:48.497668 (+ 1076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:48.499237 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.499238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.499238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.499383 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.499563 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.499564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.499564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.499606 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.499817 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:48.499818 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:48.499860 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 13:19:48.499925 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 13:19:48.500004 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 13:19:48.500512 13906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:19:48.501003 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.494412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.0.98.62:37213)
0925 13:19:48.494451 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:48.494453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:48.494492 (+ 39us) server_negotiation.cc:407] Connection header received
0925 13:19:48.494607 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:48.494608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:48.494617 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:48.494662 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:48.495073 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.495504 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.496125 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.496315 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.496364 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.498530 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:48.498531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:48.498531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:48.498538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:48.499769 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.500104 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.500105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.500105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.500158 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.500350 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.500350 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.500351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.500466 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:48.500645 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:48.500850 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 13:19:48.500921 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:19:48.506219 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.500724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37673 (local address 127.0.0.1:60064)
0925 13:19:48.500763 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.500772 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:48.500806 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:48.500906 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:48.500906 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:48.501020 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:48.501045 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.501046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.501798 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.501799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.502262 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.502263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.502580 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.502581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.502625 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.502945 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:48.502949 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:48.504012 (+ 1063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:48.505361 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.505362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.505362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.505502 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.505707 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.505708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.505708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.505757 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.505951 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:48.505951 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:48.505985 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 13:19:48.506042 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 13:19:48.506118 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 13:19:48.506374 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.500770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.0.98.60:37673)
0925 13:19:48.500800 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:48.500801 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:48.500809 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:19:48.500855 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:48.500856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:48.500865 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:48.500905 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:48.501115 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.501736 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.502327 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.502526 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.502576 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.504080 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:48.504081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:48.504081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:48.504087 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:48.505295 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.505602 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.505603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.505603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.505650 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.505806 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.505807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.505807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.505901 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:48.505951 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:48.506084 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 13:19:48.506158 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":17}
I20260925 13:19:48.507681 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.501528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.0.98.61:36019)
0925 13:19:48.501601 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:48.501603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:48.501610 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:19:48.501687 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:48.501688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:48.501697 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:48.501739 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:48.502145 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.502570 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.503167 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.503397 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.503461 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.505336 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:48.505337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:48.505337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:48.505344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:48.506723 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.507004 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.507005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.507006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.507056 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.507219 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.507220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.507220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.507320 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:48.507483 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:48.507517 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 13:19:48.507581 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:19:48.507714 13906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4: Bootstrap starting.
I20260925 13:19:48.508661 13906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:48.508989 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:19:48.508977 13906 log.cc:824] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:48.509768 13906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4: No bootstrap required, opened a new log
I20260925 13:19:48.511411 13906 raft_consensus.cc:348] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } }
I20260925 13:19:48.511534 13906 raft_consensus.cc:374] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:48.511566 13906 raft_consensus.cc:729] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49b14c6d3574f2489993c8d81fcb1f4, State: Initialized, Role: FOLLOWER
I20260925 13:19:48.511698 13906 consensus_queue.cc:260] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } }
I20260925 13:19:48.511911 13919 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } } }
I20260925 13:19:48.511977 13906 sys_catalog.cc:564] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:19:48.512603 13919 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 13:19:48.514535 13930 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:19:48.514588 13930 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:19:48.515360 13838 master_runner.cc:186] Error getting master registration for 127.0.98.62:37213: OK, instance_id { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" instance_seqno: 1790342388193824 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 13:19:48.519047 13838 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 13:19:48.519045 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.510482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60070 (local address 127.0.98.60:37673)
0925 13:19:48.510516 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:48.510517 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:48.512131 (+ 1614us) server_negotiation.cc:407] Connection header received
0925 13:19:48.512176 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:48.512177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:48.512186 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:48.512234 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:48.512846 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.513487 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.514793 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.515027 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.515086 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.516387 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:48.516389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:48.516389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:48.516396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:48.517831 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.518197 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.518198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.518198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.518256 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.518440 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.518440 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.518441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.518590 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:48.518664 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:48.518881 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 13:19:48.518961 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:19:48.520533 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.510365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.0.98.61:36019)
0925 13:19:48.510482 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:48.510484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:48.512262 (+ 1778us) server_negotiation.cc:407] Connection header received
0925 13:19:48.512341 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:48.512342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:48.512351 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:48.512377 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:48.513060 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.514025 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.516085 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.516361 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.516423 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.517783 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:48.517785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:48.517785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:48.517792 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:48.519342 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.519796 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.519797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.519797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.519855 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.520033 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.520034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.520035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.520145 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:48.520224 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:48.520371 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 13:19:48.520447 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 13:19:48.520601 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.510516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.0.98.62:37213)
0925 13:19:48.510574 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:48.510576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:48.510683 (+ 107us) server_negotiation.cc:407] Connection header received
0925 13:19:48.510739 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:48.510740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:48.510749 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:48.510793 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:48.511281 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.511826 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.512702 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.512942 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.512974 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.514322 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:48.514324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:48.514324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:48.514331 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:48.515881 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.518067 (+ 2186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.518069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.518069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.518150 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.519954 (+ 1804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.519955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.519955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.520109 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:48.520186 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:48.520393 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 13:19:48.520505 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 13:19:48.537113 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } attempt: 2
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:19:48.544585 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.537427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37673 (local address 127.0.0.1:60080)
0925 13:19:48.537484 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.537498 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:48.537566 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:48.537716 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:48.537716 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:48.537861 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:48.537903 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.537904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.538807 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.538808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.539612 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.539615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.540058 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.540059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.540117 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.540516 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:48.540521 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:48.541864 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:48.543442 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.543443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.543444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.543594 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.544009 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.544010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.544010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.544077 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.544349 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:48.544350 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:48.544391 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 13:19:48.544445 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 13:19:48.544516 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 13:19:48.544752 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.537517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60080 (local address 127.0.98.60:37673)
0925 13:19:48.537601 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:48.537604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:48.537614 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:19:48.537639 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:48.537640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:48.537650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:48.537708 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:48.537967 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.538708 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.539706 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.539993 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.540052 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.541953 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:48.541955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:48.541955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:48.541962 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:48.543373 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.543656 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.543657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.543658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.543714 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.544157 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.544158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.544158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.544288 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:48.544348 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:48.544486 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 13:19:48.544563 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 13:19:48.546135 13756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6: Bootstrap starting.
I20260925 13:19:48.546727 13756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:48.546976 13756 log.cc:824] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:48.547717 13756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6: No bootstrap required, opened a new log
I20260925 13:19:48.549105 13756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } }
I20260925 13:19:48.549218 13756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:48.549252 13756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869bb6b5f6984488b69f27549670e8e6, State: Initialized, Role: FOLLOWER
I20260925 13:19:48.549371 13756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } }
I20260925 13:19:48.549719 13756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:19:48.549746 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } } }
I20260925 13:19:48.549862 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 13:19:48.551322 13944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:19:48.551429 13944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:19:48.551949 13688 master_runner.cc:186] Error getting master registration for 127.0.98.61:36019: OK, instance_id { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" instance_seqno: 1790342388331209 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 13:19:48.554667 13688 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 13:19:48.563817 13831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } attempt: 3
Sep 25 13:19:48 dist-test-slave-t8d9 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:19:48.572268 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.564111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37673 (local address 127.0.0.1:60082)
0925 13:19:48.564160 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:48.564171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:48.564207 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:48.564381 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:48.564382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:48.564564 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:48.564616 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.564618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.565559 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.565560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.566380 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:48.566383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.567306 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.567307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:48.567371 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.567859 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:48.567864 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:48.569375 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:48.571186 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.571188 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.571188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.571411 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.571611 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:48.571612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:48.571612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:48.571678 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:48.571938 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:48.571939 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:48.571990 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 13:19:48.572061 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 13:19:48.572174 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 13:19:48.572268 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:48.564163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.0.98.60:37673)
0925 13:19:48.564217 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:48.564219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:48.564228 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:19:48.564294 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:48.564295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:48.564305 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:48.564368 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:48.564691 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.565453 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.566467 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:48.566785 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:48.566852 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:48.569465 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:48.569466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:48.569467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:48.569475 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:48.571099 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.571487 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.571488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.571488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.571548 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:48.571738 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:48.571738 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:48.571739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:48.571871 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:48.571931 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:48.572094 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 13:19:48.572170 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:19:48.574231 13831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566: Bootstrap starting.
I20260925 13:19:48.574968 13831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:48.575302 13831 log.cc:824] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:48.576169 13831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566: No bootstrap required, opened a new log
I20260925 13:19:48.578164 13831 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } }
I20260925 13:19:48.578388 13831 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:48.578437 13831 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a5a9c40e654d94a5bad833ca6f1566, State: Initialized, Role: FOLLOWER
I20260925 13:19:48.578575 13831 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } }
I20260925 13:19:48.578960 13831 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 13:19:48.580796 13958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:19:48.580849 13958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:19:48.582715 13763 master_runner.cc:263] No leader master found from master b4a5a9c40e654d94a5bad833ca6f1566
I20260925 13:19:48.582758 13763 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 13:19:48.585749 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } } }
I20260925 13:19:48.585918 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:19:48.632421 13947 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:19:48.633859 13947 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } }
I20260925 13:19:48.634732 13730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869bb6b5f6984488b69f27549670e8e6" is_pre_election: true
I20260925 13:19:48.634895 13730 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4a5a9c40e654d94a5bad833ca6f1566 in term 0.
I20260925 13:19:48.635210 13780 leader_election.cc:302] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869bb6b5f6984488b69f27549670e8e6, b4a5a9c40e654d94a5bad833ca6f1566; no voters:
I20260925 13:19:48.635756 13880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" is_pre_election: true
I20260925 13:19:48.635910 13880 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4a5a9c40e654d94a5bad833ca6f1566 in term 0.
I20260925 13:19:48.636152 13947 leader_election.cc:288] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869bb6b5f6984488b69f27549670e8e6 (127.0.98.62:37213), b49b14c6d3574f2489993c8d81fcb1f4 (127.0.98.60:37673)
I20260925 13:19:48.636292 13947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:19:48.636385 13947 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:19:48.636451 13947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:48.637374 13947 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } }
I20260925 13:19:48.637735 13730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869bb6b5f6984488b69f27549670e8e6"
I20260925 13:19:48.637822 13730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:48.638679 13730 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4a5a9c40e654d94a5bad833ca6f1566 in term 1.
I20260925 13:19:48.638912 13780 leader_election.cc:302] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869bb6b5f6984488b69f27549670e8e6, b4a5a9c40e654d94a5bad833ca6f1566; no voters:
I20260925 13:19:48.639164 13880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49b14c6d3574f2489993c8d81fcb1f4"
I20260925 13:19:48.639249 13880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:48.640066 13880 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4a5a9c40e654d94a5bad833ca6f1566 in term 1.
I20260925 13:19:48.640302 13947 leader_election.cc:288] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [CANDIDATE]: Term 1 election: Requested vote from peers 869bb6b5f6984488b69f27549670e8e6 (127.0.98.62:37213), b49b14c6d3574f2489993c8d81fcb1f4 (127.0.98.60:37673)
I20260925 13:19:48.640408 13947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:19:48.640486 13947 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [term 1 LEADER]: Becoming Leader. State: Replica: b4a5a9c40e654d94a5bad833ca6f1566, State: Running, Role: LEADER
I20260925 13:19:48.640604 13947 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } }
I20260925 13:19:48.641161 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4a5a9c40e654d94a5bad833ca6f1566. Latest consensus state: current_term: 1 leader_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } } }
I20260925 13:19:48.641299 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [sys.catalog]: This master's current role is: LEADER
I20260925 13:19:48.641583 13962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:19:48.641824 13962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:19:48.643965 13880 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4 [term 1 FOLLOWER]: Refusing update from remote peer b4a5a9c40e654d94a5bad833ca6f1566: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:19:48.644390 13960 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:19:48.646070 13919 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4a5a9c40e654d94a5bad833ca6f1566. Latest consensus state: current_term: 1 leader_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } } }
I20260925 13:19:48.646160 13919 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:19:48.646631 13730 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6 [term 1 FOLLOWER]: Refusing update from remote peer b4a5a9c40e654d94a5bad833ca6f1566: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:19:48.646934 13960 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:19:48.648524 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4a5a9c40e654d94a5bad833ca6f1566. Latest consensus state: current_term: 1 leader_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } } }
I20260925 13:19:48.648612 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:19:48.650573 13960 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } } }
I20260925 13:19:48.650661 13960 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [sys.catalog]: This master's current role is: LEADER
I20260925 13:19:48.651046 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } } }
I20260925 13:19:48.651119 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:19:48.651397 13963 mvcc.cc:205] Tried to move back new op lower bound from 7333242423883083776 to 7333242423873085440. Current Snapshot: MvccSnapshot[applied={T|T < 7333242423883083776}]
I20260925 13:19:48.652206 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } } }
I20260925 13:19:48.652283 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566 [sys.catalog]: This master's current role is: LEADER
I20260925 13:19:48.652652 13919 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bb6b5f6984488b69f27549670e8e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37213 } } peers { permanent_uuid: "b4a5a9c40e654d94a5bad833ca6f1566" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36019 } } peers { permanent_uuid: "b49b14c6d3574f2489993c8d81fcb1f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37673 } } }
I20260925 13:19:48.652745 13919 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:19:48.653378 13964 mvcc.cc:205] Tried to move back new op lower bound from 7333242423883083776 to 7333242423873085440. Current Snapshot: MvccSnapshot[applied={T|T < 7333242423883083776}]
I20260925 13:19:48.654529 13966 mvcc.cc:205] Tried to move back new op lower bound from 7333242423883083776 to 7333242423873085440. Current Snapshot: MvccSnapshot[applied={T|T < 7333242423883083776}]
I20260925 13:19:48.654752 13962 catalog_manager.cc:1347] Generated new cluster ID: cd094c6ef5774f7f8ec51fdbb64960a4
I20260925 13:19:48.654793 13962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:19:48.665014 13962 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:19:48.665876 13962 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:19:48.675989 13962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4a5a9c40e654d94a5bad833ca6f1566: Generated new TSK 0
I20260925 13:19:48.676272 13962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:19:49.515026 13930 catalog_manager.cc:1259] Loaded cluster ID: cd094c6ef5774f7f8ec51fdbb64960a4
I20260925 13:19:49.515102 13930 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4: loading cluster ID for follower catalog manager: success
I20260925 13:19:49.515908 13930 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4: acquiring CA information for follower catalog manager: success
I20260925 13:19:49.516227 13930 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b49b14c6d3574f2489993c8d81fcb1f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 13:19:49.551904 13944 catalog_manager.cc:1259] Loaded cluster ID: cd094c6ef5774f7f8ec51fdbb64960a4
I20260925 13:19:49.551981 13944 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6: loading cluster ID for follower catalog manager: success
I20260925 13:19:49.552798 13944 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6: acquiring CA information for follower catalog manager: success
I20260925 13:19:49.553071 13944 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 869bb6b5f6984488b69f27549670e8e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 13:19:49.734283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 13688
I20260925 13:19:49.738767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 13763
I20260925 13:19:49.742781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 13838
2026-09-25T13:19:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2732 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:19:49 dist-test-slave-t8d9 krb5kdc[13977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:19:49 dist-test-slave-t8d9 krb5kdc[13977](info): set up 2 sockets
Sep 25 13:19:49 dist-test-slave-t8d9 krb5kdc[13977](info): commencing operation
krb5kdc: starting...
W20260925 13:19:50.693441 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:19:50 dist-test-slave-t8d9 krb5kdc[13977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:19:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:19:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:19:50.809746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43065
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:19:50.891819 13990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:50.892035 13990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:50.892059 13990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:50.893255 13990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:19:50.893298 13990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:50.893311 13990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:50.893334 13990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:19:50.893358 13990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:19:50.894901 13990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:50.895136 13990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:50.895352 13990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:50.899873 13995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:50.899893 13996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:50.899884 13998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:50.900103 13990 server_base.cc:1034] running on GCE node
I20260925 13:19:50.900420 13990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:50.900677 13990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:50.901816 13990 hybrid_clock.cc:648] HybridClock initialized: now 1790342390901805 us; error 38 us; skew 500 ppm
Sep 25 13:19:50 dist-test-slave-t8d9 krb5kdc[13977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:50.904286 13990 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:19:50.904683 13990 webserver.cc:460] Webserver started at http://127.0.98.62:46519/ using document root <none> and password file <none>
I20260925 13:19:50.904877 13990 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:50.904917 13990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:50.905006 13990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:19:50.906082 13990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "58fe5e9448b34fc495a45cfc5d8ac966"
format_stamp: "Formatted at 2026-09-25 13:19:50 on dist-test-slave-t8d9"
server_key: "f826941b1c8acc55dd7ce428a53c061a"
server_key_iv: "b337fce85243f0877c61c156684c8cf8"
server_key_version: "encryptionkey@0"
I20260925 13:19:50.906499 13990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "58fe5e9448b34fc495a45cfc5d8ac966"
format_stamp: "Formatted at 2026-09-25 13:19:50 on dist-test-slave-t8d9"
server_key: "f826941b1c8acc55dd7ce428a53c061a"
server_key_iv: "b337fce85243f0877c61c156684c8cf8"
server_key_version: "encryptionkey@0"
I20260925 13:19:50.907768 13990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:19:50.908476 14005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:50.908661 13990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:19:50.908742 13990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "58fe5e9448b34fc495a45cfc5d8ac966"
format_stamp: "Formatted at 2026-09-25 13:19:50 on dist-test-slave-t8d9"
server_key: "f826941b1c8acc55dd7ce428a53c061a"
server_key_iv: "b337fce85243f0877c61c156684c8cf8"
server_key_version: "encryptionkey@0"
I20260925 13:19:50.908803 13990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:50.920882 13990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:50.923000 13990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:50.923136 13990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:50.926900 13990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43065
I20260925 13:19:50.926934 14057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43065 every 8 connection(s)
I20260925 13:19:50.927255 13990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:19:50.928078 14058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:50.930440 14058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966: Bootstrap starting.
I20260925 13:19:50.930982 14058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:50.931244 14058 log.cc:824] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:50.931866 14058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966: No bootstrap required, opened a new log
I20260925 13:19:50.933396 14058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fe5e9448b34fc495a45cfc5d8ac966" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43065 } }
I20260925 13:19:50.933559 14058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:50.933609 14058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58fe5e9448b34fc495a45cfc5d8ac966, State: Initialized, Role: FOLLOWER
I20260925 13:19:50.933724 14058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fe5e9448b34fc495a45cfc5d8ac966" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43065 } }
I20260925 13:19:50.933779 14058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:19:50.933804 14058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:19:50.933833 14058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:50.934324 14058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fe5e9448b34fc495a45cfc5d8ac966" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43065 } }
I20260925 13:19:50.934383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 13990
I20260925 13:19:50.934473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:19:50.934458 14058 leader_election.cc:302] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58fe5e9448b34fc495a45cfc5d8ac966; no voters:
I20260925 13:19:50.934609 14058 leader_election.cc:288] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:19:50.934671 14061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:19:50.934723 392 external_mini_cluster.cc:1428] Setting key d20cbe3136a0e67ff756ce028f162c30
I20260925 13:19:50.934856 14058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:19:50.935101 14061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [term 1 LEADER]: Becoming Leader. State: Replica: 58fe5e9448b34fc495a45cfc5d8ac966, State: Running, Role: LEADER
I20260925 13:19:50.935206 14061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fe5e9448b34fc495a45cfc5d8ac966" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43065 } }
I20260925 13:19:50.935554 14062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58fe5e9448b34fc495a45cfc5d8ac966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fe5e9448b34fc495a45cfc5d8ac966" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43065 } } }
I20260925 13:19:50.935635 14062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [sys.catalog]: This master's current role is: LEADER
I20260925 13:19:50.935783 14063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58fe5e9448b34fc495a45cfc5d8ac966. Latest consensus state: current_term: 1 leader_uuid: "58fe5e9448b34fc495a45cfc5d8ac966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fe5e9448b34fc495a45cfc5d8ac966" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43065 } } }
I20260925 13:19:50.935832 14072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:19:50.935914 14063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966 [sys.catalog]: This master's current role is: LEADER
I20260925 13:19:50.936388 14072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:19:50.938004 14072 catalog_manager.cc:1347] Generated new cluster ID: 9cde3ad41ab4488e9114b421f6017a21
I20260925 13:19:50.938059 14072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:19:50 dist-test-slave-t8d9 krb5kdc[13977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:50.945812 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:50.935336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.0.98.62:43065)
0925 13:19:50.936202 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:50.936212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:50.936257 (+ 45us) server_negotiation.cc:407] Connection header received
0925 13:19:50.936375 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:50.936383 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:50.936440 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:50.937318 (+ 878us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:50.937408 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:50.938565 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:50.939383 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:50.939817 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:50.939916 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:50.942162 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:50.942166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:50.942167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:50.942180 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:50.944282 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:50.944620 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:50.944623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:50.944624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:50.944715 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:50.944894 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:50.944895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:50.944895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:50.945135 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:50.945304 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:50.945416 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 13:19:50.945591 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":82,"threads_started":1}
I20260925 13:19:50.946693 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:19:50.971326 14072 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:19:50.971858 14072 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:19:50.975050 14072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58fe5e9448b34fc495a45cfc5d8ac966: Generated new TSK 0
I20260925 13:19:50.975240 14072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:19:51.000481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 13990
2026-09-25T13:19:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1256 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:19:51 dist-test-slave-t8d9 krb5kdc[14085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:19:51 dist-test-slave-t8d9 krb5kdc[14085](info): set up 2 sockets
Sep 25 13:19:51 dist-test-slave-t8d9 krb5kdc[14085](info): commencing operation
krb5kdc: starting...
W20260925 13:19:51.945367 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:19:52 dist-test-slave-t8d9 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:19:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:19:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:19:52.060415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42013
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:19:52.139041 14098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:52.139253 14098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:52.139276 14098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:52.140409 14098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:19:52.140451 14098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:52.140465 14098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:52.140487 14098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:19:52.140508 14098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:19:52.141990 14098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:52.142220 14098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:52.142422 14098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:52.146967 14106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:52.146951 14104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:52.146953 14103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:52.147157 14098 server_base.cc:1034] running on GCE node
I20260925 13:19:52.147388 14098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:52.147701 14098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:52.148860 14098 hybrid_clock.cc:648] HybridClock initialized: now 1790342392148842 us; error 32 us; skew 500 ppm
Sep 25 13:19:52 dist-test-slave-t8d9 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:52.151292 14098 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:19:52.151755 14098 webserver.cc:460] Webserver started at http://127.0.98.62:38837/ using document root <none> and password file <none>
I20260925 13:19:52.151957 14098 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:52.152004 14098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:52.152128 14098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:19:52.153146 14098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "ad83e0eda75740fbbed1bda0850e566d"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "ec32d488ae11ad703efaa59836f5facd"
server_key_iv: "4a1d2a5fe0550130ca31c61da651761a"
server_key_version: "encryptionkey@0"
I20260925 13:19:52.153472 14098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "ad83e0eda75740fbbed1bda0850e566d"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "ec32d488ae11ad703efaa59836f5facd"
server_key_iv: "4a1d2a5fe0550130ca31c61da651761a"
server_key_version: "encryptionkey@0"
I20260925 13:19:52.154702 14098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:19:52.155377 14113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:52.155525 14098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:19:52.155613 14098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "ad83e0eda75740fbbed1bda0850e566d"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "ec32d488ae11ad703efaa59836f5facd"
server_key_iv: "4a1d2a5fe0550130ca31c61da651761a"
server_key_version: "encryptionkey@0"
I20260925 13:19:52.155673 14098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:52.166190 14098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:52.168032 14098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:52.168170 14098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:52.171936 14098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42013
I20260925 13:19:52.171978 14165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42013 every 8 connection(s)
I20260925 13:19:52.172302 14098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:19:52.173285 14166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:52.174997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 14098
I20260925 13:19:52.175079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:19:52.175247 392 external_mini_cluster.cc:1428] Setting key c618fea2843b875a14d08fb21cdfd0e7
I20260925 13:19:52.175711 14166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: Bootstrap starting.
I20260925 13:19:52.176415 14166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:52.176723 14166 log.cc:824] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:52.177390 14166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: No bootstrap required, opened a new log
I20260925 13:19:52.179147 14166 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260925 13:19:52.179282 14166 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:52.179312 14166 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad83e0eda75740fbbed1bda0850e566d, State: Initialized, Role: FOLLOWER
I20260925 13:19:52.179436 14166 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260925 13:19:52.179512 14166 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:19:52.179543 14166 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:19:52.179589 14166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:52.180095 14166 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260925 13:19:52.180186 14166 leader_election.cc:302] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad83e0eda75740fbbed1bda0850e566d; no voters:
I20260925 13:19:52.180339 14166 leader_election.cc:288] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:19:52.180410 14171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:19:52.180549 14171 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 1 LEADER]: Becoming Leader. State: Replica: ad83e0eda75740fbbed1bda0850e566d, State: Running, Role: LEADER
I20260925 13:19:52.180624 14166 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:19:52.180699 14171 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260925 13:19:52.181043 14171 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad83e0eda75740fbbed1bda0850e566d. Latest consensus state: current_term: 1 leader_uuid: "ad83e0eda75740fbbed1bda0850e566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } } }
I20260925 13:19:52.181054 14173 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad83e0eda75740fbbed1bda0850e566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } } }
I20260925 13:19:52.181181 14171 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [sys.catalog]: This master's current role is: LEADER
I20260925 13:19:52.181192 14173 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [sys.catalog]: This master's current role is: LEADER
Sep 25 13:19:52 dist-test-slave-t8d9 krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260925 13:19:52.182516 14186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:19:52.182559 14186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:19:52.182651 14187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:19:52.182834 14187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:19:52.185415 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.175818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38260 (local address 127.0.98.62:42013)
0925 13:19:52.176019 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:52.176028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:52.176065 (+ 37us) server_negotiation.cc:407] Connection header received
0925 13:19:52.176637 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:52.176647 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:52.176714 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:52.176864 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:52.177567 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.178464 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.179205 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.179702 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.179798 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.182378 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:52.182382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:52.182382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:52.182396 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:52.184093 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:52.184405 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:52.184409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:52.184409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:52.184490 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:52.184666 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:52.184666 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:52.184667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:52.184882 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:52.185039 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:52.185115 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 13:19:52.185258 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":97,"threads_started":1}
I20260925 13:19:52.185904 14187 catalog_manager.cc:1347] Generated new cluster ID: f281f850dc7441d386844db9e510c738
I20260925 13:19:52.185956 14187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:19:52.195176 14187 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:19:52.195720 14187 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:19:52.206351 14187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: Generated new TSK 0
I20260925 13:19:52.206537 14187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:19:52.233134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42013
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:19:52.315001 14192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:52.315228 14192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:52.315253 14192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:52.316505 14192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:52.316558 14192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:52.316593 14192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:19:52.318292 14192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:52.318548 14192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:52.318785 14192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:52.323515 14198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:52.323616 14200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:52.323521 14197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:52.323930 14192 server_base.cc:1034] running on GCE node
I20260925 13:19:52.324115 14192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:52.324329 14192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:52.325495 14192 hybrid_clock.cc:648] HybridClock initialized: now 1790342392325476 us; error 28 us; skew 500 ppm
Sep 25 13:19:52 dist-test-slave-t8d9 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:52.328059 14192 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:19:52.328490 14192 webserver.cc:460] Webserver started at http://127.0.98.1:36453/ using document root <none> and password file <none>
I20260925 13:19:52.328744 14192 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:52.328792 14192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:52.328918 14192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:19:52.330008 14192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "9194a39b177a4b83a94683693cd20485"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "7f6a7310e4d4496a76d573d6697ae7e5"
server_key_iv: "41d12f7b55c79896bfba991157d42c5e"
server_key_version: "encryptionkey@0"
I20260925 13:19:52.330307 14192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9194a39b177a4b83a94683693cd20485"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "7f6a7310e4d4496a76d573d6697ae7e5"
server_key_iv: "41d12f7b55c79896bfba991157d42c5e"
server_key_version: "encryptionkey@0"
I20260925 13:19:52.331550 14192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:19:52.332329 14207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:52.332518 14192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:19:52.332600 14192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "9194a39b177a4b83a94683693cd20485"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "7f6a7310e4d4496a76d573d6697ae7e5"
server_key_iv: "41d12f7b55c79896bfba991157d42c5e"
server_key_version: "encryptionkey@0"
I20260925 13:19:52.332665 14192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:52.341281 14192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:52.343226 14192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:52.343361 14192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:52.343602 14192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:52.343919 14192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:19:52.343952 14192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:52.343981 14192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:19:52.344007 14192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:52.350274 14192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45323
I20260925 13:19:52.350327 14320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45323 every 8 connection(s)
I20260925 13:19:52.350617 14192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 13:19:52 dist-test-slave-t8d9 krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:52.357877 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 14192
I20260925 13:19:52.357982 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:19:52.358178 392 external_mini_cluster.cc:1428] Setting key 5540593acefe63405cff59fc4350cdcf
I20260925 13:19:52.359935 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.351303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52969 (local address 127.0.98.62:42013)
0925 13:19:52.351371 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:52.351373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:52.351551 (+ 178us) server_negotiation.cc:407] Connection header received
0925 13:19:52.352087 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:52.352088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:52.352100 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:52.352153 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:52.353095 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.353659 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.354657 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.354866 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.354924 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.357107 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:52.357109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:52.357109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:52.357115 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:52.358626 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:52.358994 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:52.358995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:52.358996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:52.359054 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:52.359318 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:52.359319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:52.359319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:52.359433 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:52.359488 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:52.359759 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 13:19:52.359847 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 13:19:52.360219 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.351259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:52969)
0925 13:19:52.351425 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:52.351468 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:52.351963 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:52.352258 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:52.352262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:52.352803 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:52.352996 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.353001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.353745 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.353746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.354572 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.354574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.354940 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.354941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.355068 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.355619 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:52.355627 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:52.357015 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:52.358695 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:52.358704 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:52.358706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:52.358926 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:52.359108 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:52.359109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:52.359109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:52.359240 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:52.359490 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:52.359497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:52.359639 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 13:19:52.359728 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 13:19:52.359888 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260925 13:19:52.360728 14321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42013
I20260925 13:19:52.360813 14321 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:52.360984 14321 heartbeater.cc:511] Master 127.0.98.62:42013 requested a full tablet report, sending...
I20260925 13:19:52.361490 14130 ts_manager.cc:194] Registered new tserver with Master: 9194a39b177a4b83a94683693cd20485 (127.0.98.1:45323)
I20260925 13:19:52.362116 14130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52969
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:19:52.382164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42013
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:19:52.463670 14326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:52.463891 14326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:52.463914 14326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:52.465022 14326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:52.465057 14326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:52.465090 14326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:19:52.466599 14326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:52.466792 14326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:52.466975 14326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:52.471801 14331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:52.471903 14326 server_base.cc:1034] running on GCE node
W20260925 13:19:52.471819 14334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:52.471807 14332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:52.472205 14326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:52.472435 14326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:52.473594 14326 hybrid_clock.cc:648] HybridClock initialized: now 1790342392473573 us; error 29 us; skew 500 ppm
Sep 25 13:19:52 dist-test-slave-t8d9 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:52.475979 14326 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:19:52.476435 14326 webserver.cc:460] Webserver started at http://127.0.98.2:37031/ using document root <none> and password file <none>
I20260925 13:19:52.476611 14326 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:52.476648 14326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:52.476738 14326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:19:52.477780 14326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "d286e9d29072438da6a94fcb457836ef"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "28d01c50ff025189c816a11b8a576ceb"
server_key_iv: "e5babdaf2945742eb38b4ab3230dcab6"
server_key_version: "encryptionkey@0"
I20260925 13:19:52.478081 14326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d286e9d29072438da6a94fcb457836ef"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "28d01c50ff025189c816a11b8a576ceb"
server_key_iv: "e5babdaf2945742eb38b4ab3230dcab6"
server_key_version: "encryptionkey@0"
I20260925 13:19:52.479236 14326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:19:52.479903 14341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:52.480074 14326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:19:52.480152 14326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "d286e9d29072438da6a94fcb457836ef"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "28d01c50ff025189c816a11b8a576ceb"
server_key_iv: "e5babdaf2945742eb38b4ab3230dcab6"
server_key_version: "encryptionkey@0"
I20260925 13:19:52.480208 14326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:52.491878 14326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:52.493991 14326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:52.494104 14326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:52.494289 14326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:52.494549 14326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:19:52.494578 14326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:52.494599 14326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:19:52.494616 14326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:52.499790 14326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45487
I20260925 13:19:52.499830 14454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45487 every 8 connection(s)
I20260925 13:19:52.500131 14326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:19:52 dist-test-slave-t8d9 krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:52.506870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 14326
I20260925 13:19:52.506955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:19:52.507143 392 external_mini_cluster.cc:1428] Setting key 02fa367ad5287ba3e23c8b31a07d46c1
I20260925 13:19:52.509003 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.500787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32951 (local address 127.0.98.62:42013)
0925 13:19:52.500856 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:52.500858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:52.501034 (+ 176us) server_negotiation.cc:407] Connection header received
0925 13:19:52.501477 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:52.501478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:52.501488 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:52.501558 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:52.502511 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.503026 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.503996 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.504186 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.504237 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.506230 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:52.506231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:52.506231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:52.506238 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:52.507697 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:52.508147 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:52.508148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:52.508148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:52.508207 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:52.508449 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:52.508450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:52.508450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:52.508542 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:52.508585 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:52.508876 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 13:19:52.508940 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 13:19:52.509251 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.500758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:32951)
0925 13:19:52.500911 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:52.500948 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:52.501401 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:52.501641 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:52.501645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:52.502205 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:52.502398 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.502403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.503109 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.503110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.503922 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.503924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.504250 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.504250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.504343 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.504783 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:52.504790 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:52.506146 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:52.507780 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:52.507788 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:52.507790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:52.508062 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:52.508262 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:52.508263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:52.508263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:52.508395 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:52.508593 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:52.508599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:52.508756 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 13:19:52.508862 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 13:19:52.509016 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260925 13:19:52.509704 14455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42013
I20260925 13:19:52.509814 14455 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:52.510000 14455 heartbeater.cc:511] Master 127.0.98.62:42013 requested a full tablet report, sending...
I20260925 13:19:52.510372 14130 ts_manager.cc:194] Registered new tserver with Master: d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487)
I20260925 13:19:52.510748 14130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32951
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:19:52.532891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42013
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:19:52.614382 14460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:52.614621 14460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:52.614647 14460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:52.615831 14460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:52.615868 14460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:52.615895 14460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:19:52.617360 14460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:52.617620 14460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:52.617832 14460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:52.622659 14468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:52.622812 14460 server_base.cc:1034] running on GCE node
W20260925 13:19:52.622669 14466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:52.622655 14465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:52.623077 14460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:52.623318 14460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:52.624462 14460 hybrid_clock.cc:648] HybridClock initialized: now 1790342392624446 us; error 34 us; skew 500 ppm
Sep 25 13:19:52 dist-test-slave-t8d9 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:52.627044 14460 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:19:52.627485 14460 webserver.cc:460] Webserver started at http://127.0.98.3:40831/ using document root <none> and password file <none>
I20260925 13:19:52.627723 14460 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:52.627777 14460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:52.627902 14460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:19:52.628974 14460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "c20564e854c642b5a313d613226ca35d"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "1a2cc5fb65a6e55c2755006a43e8829f"
server_key_iv: "dba6aa347acb17e04b8209fe720a1a24"
server_key_version: "encryptionkey@0"
I20260925 13:19:52.629304 14460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c20564e854c642b5a313d613226ca35d"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "1a2cc5fb65a6e55c2755006a43e8829f"
server_key_iv: "dba6aa347acb17e04b8209fe720a1a24"
server_key_version: "encryptionkey@0"
I20260925 13:19:52.630651 14460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 13:19:52.631493 14475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:52.631681 14460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:19:52.631768 14460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "c20564e854c642b5a313d613226ca35d"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "1a2cc5fb65a6e55c2755006a43e8829f"
server_key_iv: "dba6aa347acb17e04b8209fe720a1a24"
server_key_version: "encryptionkey@0"
I20260925 13:19:52.631839 14460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:52.650009 14460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:52.652098 14460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:52.652256 14460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:52.652520 14460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:52.652845 14460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:19:52.652881 14460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:52.652916 14460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:19:52.652944 14460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:52.659240 14460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34749
I20260925 13:19:52.659289 14588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34749 every 8 connection(s)
I20260925 13:19:52.659639 14460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:19:52 dist-test-slave-t8d9 krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:52.667757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 14460
I20260925 13:19:52.667848 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:19:52.668026 392 external_mini_cluster.cc:1428] Setting key 3006efd14f8ccf760d7f2a4069c2a8b5
I20260925 13:19:52.668774 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.660402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56203 (local address 127.0.98.62:42013)
0925 13:19:52.660456 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:52.660458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:52.660649 (+ 191us) server_negotiation.cc:407] Connection header received
0925 13:19:52.661167 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:52.661168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:52.661181 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:52.661229 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:52.662307 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.662830 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.663836 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.664029 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.664086 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.666205 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:52.666207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:52.666208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:52.666216 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:52.667409 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:52.667803 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:52.667804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:52.667805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:52.667871 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:52.668106 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:52.668107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:52.668107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:52.668222 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:52.668280 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:52.668586 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 13:19:52.668682 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 13:19:52.668954 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.660378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:56203)
0925 13:19:52.660526 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:52.660566 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:52.661065 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:52.661316 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:52.661320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:52.661998 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:52.662212 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.662218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.662920 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.662921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.663739 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.663741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.664102 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.664103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.664230 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.664717 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:52.664725 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:52.666114 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:52.667498 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:52.667504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:52.667507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:52.667729 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:52.667932 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:52.667933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:52.667933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:52.668045 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:52.668283 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:52.668289 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:52.668447 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 13:19:52.668560 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 13:19:52.668728 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260925 13:19:52.669649 14589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42013
I20260925 13:19:52.669749 14589 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:52.669894 14589 heartbeater.cc:511] Master 127.0.98.62:42013 requested a full tablet report, sending...
I20260925 13:19:52.670225 14130 ts_manager.cc:194] Registered new tserver with Master: c20564e854c642b5a313d613226ca35d (127.0.98.3:34749)
I20260925 13:19:52.670565 14130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56203
I20260925 13:19:52.679750 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:19:52.686939 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.681124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.0.98.62:42013)
0925 13:19:52.681172 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:52.681174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:52.681302 (+ 128us) server_negotiation.cc:407] Connection header received
0925 13:19:52.681343 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:52.681344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:52.681353 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:52.681401 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:52.681979 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.682558 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.683316 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.683496 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.683546 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.684669 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:52.684670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:52.684670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:52.684676 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:52.685824 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:52.686126 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:52.686127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:52.686128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:52.686179 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:52.686430 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:52.686431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:52.686431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:52.686594 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:52.686684 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:52.686791 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 13:19:52.686859 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:19:52.688431 14130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:19:52.690016 14130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:19:52.696327 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.692239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45323 (local address 127.0.0.1:36400)
0925 13:19:52.692532 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:52.692576 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:52.692803 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:52.693058 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:52.693058 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:52.693060 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:52.693127 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.693129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.694026 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.694027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.695159 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.695161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.696001 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.696002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.696097 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.696112 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 13:19:52.696184 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 13:19:52.696246 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":3,"thread_start_us":63,"threads_started":1}
I20260925 13:19:52.696337 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.692345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.0.98.1:45323)
0925 13:19:52.692570 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:52.692575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:52.692642 (+ 67us) server_negotiation.cc:407] Connection header received
0925 13:19:52.692891 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:52.692893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:52.692906 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:52.692958 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:52.693206 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.693938 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.695249 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.695912 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.695980 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.696042 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:52.696208 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 13:19:52.696250 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
I20260925 13:19:52.696610 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.692240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34749 (local address 127.0.0.1:44828)
0925 13:19:52.692622 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:52.692632 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:52.692710 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:52.693014 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:52.693017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:52.693020 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:52.693089 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.693091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.694438 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.694439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.695422 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.695423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.696331 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.696332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.696393 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.696397 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:19:52.696504 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 13:19:52.696541 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":24,"thread_start_us":68,"threads_started":1}
I20260925 13:19:52.696827 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.692347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.0.98.3:34749)
0925 13:19:52.692607 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:52.692610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:52.692703 (+ 93us) server_negotiation.cc:407] Connection header received
0925 13:19:52.692811 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:52.692813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:52.692840 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:52.692917 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:52.693178 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.694320 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.695496 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.696174 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.696242 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.696295 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:52.696685 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 13:19:52.696739 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":86,"threads_started":1}
I20260925 13:19:52.697113 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.692445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45487 (local address 127.0.0.1:56752)
0925 13:19:52.692686 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:52.692694 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:52.692734 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:52.693048 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:52.693049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:52.693051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:52.693117 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.693119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.694454 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.694455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.695573 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.695576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.696880 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.696881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.696960 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.696963 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:19:52.697001 (+ 38us) client_negotiation.cc:240] Negotiation successful
0925 13:19:52.697037 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":4,"thread_start_us":31,"threads_started":1}
I20260925 13:19:52.697099 14254 tablet_service.cc:1467] Processing CreateTablet for tablet 96ac01a3a5644aa7adc7a7f22aa3481f (DEFAULT_TABLE table=test-table [id=5f888484d07d480cb01cfa07bc2ddaad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:19:52.697315 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.692583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.0.98.2:45487)
0925 13:19:52.692824 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:52.692829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:52.692838 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:19:52.692875 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:52.692877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:52.692893 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:52.692977 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:52.693183 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.694345 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.695664 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.696349 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.696863 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.697054 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:52.697079 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 13:19:52.697122 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
I20260925 13:19:52.697503 14254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ac01a3a5644aa7adc7a7f22aa3481f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:52.698189 14523 tablet_service.cc:1467] Processing CreateTablet for tablet 96ac01a3a5644aa7adc7a7f22aa3481f (DEFAULT_TABLE table=test-table [id=5f888484d07d480cb01cfa07bc2ddaad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:19:52.698556 14523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ac01a3a5644aa7adc7a7f22aa3481f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:52.701036 14603 tablet_bootstrap.cc:492] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: Bootstrap starting.
I20260925 13:19:52.701491 14389 tablet_service.cc:1467] Processing CreateTablet for tablet 96ac01a3a5644aa7adc7a7f22aa3481f (DEFAULT_TABLE table=test-table [id=5f888484d07d480cb01cfa07bc2ddaad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:19:52.701782 14603 tablet_bootstrap.cc:654] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:52.701879 14389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ac01a3a5644aa7adc7a7f22aa3481f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:52.702121 14604 tablet_bootstrap.cc:492] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d: Bootstrap starting.
I20260925 13:19:52.702394 14603 log.cc:824] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:52.702759 14604 tablet_bootstrap.cc:654] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:52.703260 14604 log.cc:824] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:52.703320 14603 tablet_bootstrap.cc:492] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: No bootstrap required, opened a new log
I20260925 13:19:52.703373 14603 ts_tablet_manager.cc:1397] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:19:52.704002 14604 tablet_bootstrap.cc:492] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d: No bootstrap required, opened a new log
I20260925 13:19:52.704066 14604 ts_tablet_manager.cc:1397] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:19:52.704916 14607 tablet_bootstrap.cc:492] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Bootstrap starting.
I20260925 13:19:52.705358 14603 raft_consensus.cc:348] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:52.705515 14607 tablet_bootstrap.cc:654] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:52.705564 14603 raft_consensus.cc:374] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:52.705605 14603 raft_consensus.cc:729] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9194a39b177a4b83a94683693cd20485, State: Initialized, Role: FOLLOWER
I20260925 13:19:52.705749 14603 consensus_queue.cc:260] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:52.706032 14603 ts_tablet_manager.cc:1428] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 13:19:52.706023 14607 log.cc:824] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:52.706192 14321 heartbeater.cc:503] Master 127.0.98.62:42013 was elected leader, sending a full tablet report...
I20260925 13:19:52.706265 14604 raft_consensus.cc:348] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:52.706393 14604 raft_consensus.cc:374] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:52.706434 14604 raft_consensus.cc:729] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20564e854c642b5a313d613226ca35d, State: Initialized, Role: FOLLOWER
I20260925 13:19:52.706547 14604 consensus_queue.cc:260] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:52.706832 14604 ts_tablet_manager.cc:1428] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:19:52.706952 14589 heartbeater.cc:503] Master 127.0.98.62:42013 was elected leader, sending a full tablet report...
I20260925 13:19:52.707212 14607 tablet_bootstrap.cc:492] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: No bootstrap required, opened a new log
I20260925 13:19:52.707271 14607 ts_tablet_manager.cc:1397] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:19:52.708817 14607 raft_consensus.cc:348] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:52.708948 14607 raft_consensus.cc:374] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:52.708978 14607 raft_consensus.cc:729] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d286e9d29072438da6a94fcb457836ef, State: Initialized, Role: FOLLOWER
I20260925 13:19:52.709084 14607 consensus_queue.cc:260] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:52.709326 14607 ts_tablet_manager.cc:1428] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:19:52.709385 14455 heartbeater.cc:503] Master 127.0.98.62:42013 was elected leader, sending a full tablet report...
W20260925 13:19:52.750666 14456 tablet.cc:2369] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:19:52.851292 14322 tablet.cc:2369] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:19:52.910202 14590 tablet.cc:2369] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:19:52.919526 14608 raft_consensus.cc:482] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:19:52.919667 14608 raft_consensus.cc:504] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:52.920006 14608 leader_election.cc:288] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c20564e854c642b5a313d613226ca35d (127.0.98.3:34749), d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487)
I20260925 13:19:52.922911 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.920169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34749 (local address 127.0.98.1:46455)
0925 13:19:52.920287 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:52.920298 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:52.920357 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:52.920482 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:52.920483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:52.920485 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:52.920565 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.920568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.921257 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.921258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.922127 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.922129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.922687 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.922688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.922752 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.922756 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:19:52.922805 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 13:19:52.922841 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":20,"thread_start_us":68,"threads_started":1}
I20260925 13:19:52.922926 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.920248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46455 (local address 127.0.98.3:34749)
0925 13:19:52.920298 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:52.920300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:52.920365 (+ 65us) server_negotiation.cc:407] Connection header received
0925 13:19:52.920401 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:52.920402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:52.920413 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:52.920466 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:52.920632 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.921158 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.922220 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.922611 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.922662 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.922670 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:52.922822 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 13:19:52.922859 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:19:52.923027 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.920238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34959 (local address 127.0.98.2:45487)
0925 13:19:52.920294 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:52.920296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:52.920398 (+ 102us) server_negotiation.cc:407] Connection header received
0925 13:19:52.920449 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:52.920450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:52.920462 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:52.920511 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:52.920813 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.921321 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.922305 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.922705 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.922773 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.922783 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:52.922927 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 13:19:52.922968 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:19:52.923173 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.920169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45487 (local address 127.0.98.1:34959)
0925 13:19:52.920314 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:52.920324 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:52.920397 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:52.920646 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:52.920646 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:52.920648 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:52.920715 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.920717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.921404 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.921404 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.922219 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:52.922222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.922780 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.922781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:52.922853 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.922856 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:19:52.922908 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 13:19:52.922942 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260925 13:19:52.923410 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ac01a3a5644aa7adc7a7f22aa3481f" candidate_uuid: "9194a39b177a4b83a94683693cd20485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20564e854c642b5a313d613226ca35d" is_pre_election: true
I20260925 13:19:52.923446 14409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ac01a3a5644aa7adc7a7f22aa3481f" candidate_uuid: "9194a39b177a4b83a94683693cd20485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d286e9d29072438da6a94fcb457836ef" is_pre_election: true
I20260925 13:19:52.923553 14543 raft_consensus.cc:2454] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9194a39b177a4b83a94683693cd20485 in term 0.
I20260925 13:19:52.923583 14409 raft_consensus.cc:2454] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9194a39b177a4b83a94683693cd20485 in term 0.
I20260925 13:19:52.923774 14208 leader_election.cc:302] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9194a39b177a4b83a94683693cd20485, c20564e854c642b5a313d613226ca35d; no voters:
I20260925 13:19:52.923942 14608 raft_consensus.cc:2781] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:19:52.923997 14608 raft_consensus.cc:482] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:19:52.924019 14608 raft_consensus.cc:3037] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:52.924717 14608 raft_consensus.cc:504] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:52.924896 14608 leader_election.cc:288] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 1 election: Requested vote from peers c20564e854c642b5a313d613226ca35d (127.0.98.3:34749), d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487)
I20260925 13:19:52.924966 14409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ac01a3a5644aa7adc7a7f22aa3481f" candidate_uuid: "9194a39b177a4b83a94683693cd20485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d286e9d29072438da6a94fcb457836ef"
I20260925 13:19:52.925004 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ac01a3a5644aa7adc7a7f22aa3481f" candidate_uuid: "9194a39b177a4b83a94683693cd20485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20564e854c642b5a313d613226ca35d"
I20260925 13:19:52.925047 14409 raft_consensus.cc:3037] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:52.925094 14543 raft_consensus.cc:3037] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:52.925789 14543 raft_consensus.cc:2454] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9194a39b177a4b83a94683693cd20485 in term 1.
I20260925 13:19:52.925853 14409 raft_consensus.cc:2454] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9194a39b177a4b83a94683693cd20485 in term 1.
I20260925 13:19:52.926033 14209 leader_election.cc:302] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9194a39b177a4b83a94683693cd20485, d286e9d29072438da6a94fcb457836ef; no voters:
I20260925 13:19:52.926156 14608 raft_consensus.cc:2781] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:19:52.926324 14608 raft_consensus.cc:686] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [term 1 LEADER]: Becoming Leader. State: Replica: 9194a39b177a4b83a94683693cd20485, State: Running, Role: LEADER
I20260925 13:19:52.926431 14608 consensus_queue.cc:237] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:52.927224 14130 catalog_manager.cc:5495] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9194a39b177a4b83a94683693cd20485 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9194a39b177a4b83a94683693cd20485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } health_report { overall_health: UNKNOWN } } }
I20260925 13:19:52.978807 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.976625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.0.98.1:45323)
0925 13:19:52.976679 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:52.976681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:52.976690 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:19:52.976728 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:52.976729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:52.976749 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:52.976811 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:19:52.976928 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.977386 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.978114 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:52.978291 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:52.978335 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:52.978501 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:19:52.978638 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:19:52.978714 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:52.978730 (+ 16us) server_negotiation.cc:299] Negotiation successful
0925 13:19:52.978748 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:19:52.980674 14543 raft_consensus.cc:1261] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d [term 1 FOLLOWER]: Refusing update from remote peer 9194a39b177a4b83a94683693cd20485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:19:52.980659 14409 raft_consensus.cc:1261] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef [term 1 FOLLOWER]: Refusing update from remote peer 9194a39b177a4b83a94683693cd20485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:19:52.981005 14608 consensus_queue.cc:1035] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:19:52.981161 14614 consensus_queue.cc:1035] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:19:52.984256 14618 mvcc.cc:205] Tried to move back new op lower bound from 7333242441646022656 to 7333242441427509248. Current Snapshot: MvccSnapshot[applied={T|T < 7333242441646022656}]
I20260925 13:19:52.986505 14130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:19:52.986591 14130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:19:52.987238 14130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: Sending DeleteTablet for 3 replicas of tablet 96ac01a3a5644aa7adc7a7f22aa3481f
I20260925 13:19:52.987555 14389 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ac01a3a5644aa7adc7a7f22aa3481f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:19:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56752
I20260925 13:19:52.987593 14523 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ac01a3a5644aa7adc7a7f22aa3481f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:19:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44828
I20260925 13:19:52.987777 14254 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ac01a3a5644aa7adc7a7f22aa3481f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:19:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36400
I20260925 13:19:52.987954 14629 tablet_replica.cc:330] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: stopping tablet replica
I20260925 13:19:52.987959 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 14192
I20260925 13:19:52.992375 14628 tablet_replica.cc:330] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: stopping tablet replica
W20260925 13:19:52.992385 14115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45323 (error 108)
I20260925 13:19:52.992481 14628 raft_consensus.cc:2229] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:19:52.992501 14627 tablet_replica.cc:330] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d: stopping tablet replica
I20260925 13:19:52.992573 14627 raft_consensus.cc:2229] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:19:52.992600 14628 raft_consensus.cc:2258] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:19:52.992661 14627 raft_consensus.cc:2258] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:19:52.992851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 14326
I20260925 13:19:52.992969 14628 ts_tablet_manager.cc:1905] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:19:52.993582 14627 ts_tablet_manager.cc:1905] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:19:52.993736 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.993628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45323 (local address 127.0.0.1:36418)
0925 13:19:52.993671 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:52.993706 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 13:19:52.993829 14115 catalog_manager.cc:4584] TS 9194a39b177a4b83a94683693cd20485 (127.0.98.1:45323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96ac01a3a5644aa7adc7a7f22aa3481f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45323: connect: Connection refused (error 111)
I20260925 13:19:52.995312 14627 ts_tablet_manager.cc:1918] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:19:52.995389 14627 log.cc:1196] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/96ac01a3a5644aa7adc7a7f22aa3481f
I20260925 13:19:52.995720 14627 ts_tablet_manager.cc:1939] T 96ac01a3a5644aa7adc7a7f22aa3481f P c20564e854c642b5a313d613226ca35d: Deleting consensus metadata
I20260925 13:19:52.996168 14114 catalog_manager.cc:4841] TS c20564e854c642b5a313d613226ca35d (127.0.98.3:34749): tablet 96ac01a3a5644aa7adc7a7f22aa3481f (table test-table [id=5f888484d07d480cb01cfa07bc2ddaad]) successfully deleted
I20260925 13:19:52.996877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 14460
I20260925 13:19:52.996933 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:52.996831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45487 (local address 127.0.0.1:56766)
0925 13:19:52.996878 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:52.996907 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 13:19:53.001350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 14098
I20260925 13:19:53.005576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42013
--webserver_interface=127.0.98.62
--webserver_port=38837
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:19:53.084448 14631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:53.084674 14631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:53.084695 14631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:53.085906 14631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:19:53.085945 14631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:53.085958 14631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:53.085969 14631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:19:53.085979 14631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:19:53.087438 14631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:53.087667 14631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:53.087884 14631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:53.095600 14637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:53.095731 14631 server_base.cc:1034] running on GCE node
W20260925 13:19:53.095610 14640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:53.095610 14638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:53.096103 14631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:53.096377 14631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:53.097548 14631 hybrid_clock.cc:648] HybridClock initialized: now 1790342393097509 us; error 30 us; skew 500 ppm
Sep 25 13:19:53 dist-test-slave-t8d9 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:53.100008 14631 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:19:53.100455 14631 webserver.cc:460] Webserver started at http://127.0.98.62:38837/ using document root <none> and password file <none>
I20260925 13:19:53.100651 14631 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:53.100701 14631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:53.102029 14631 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:19:53.102646 14647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:53.102814 14631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:19:53.102900 14631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "ad83e0eda75740fbbed1bda0850e566d"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "ec32d488ae11ad703efaa59836f5facd"
server_key_iv: "4a1d2a5fe0550130ca31c61da651761a"
server_key_version: "encryptionkey@0"
I20260925 13:19:53.103298 14631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:53.118623 14631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:53.120882 14631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:53.121026 14631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:53.124838 14631 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42013
I20260925 13:19:53.124882 14699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42013 every 8 connection(s)
I20260925 13:19:53.125264 14631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:19:53.126863 14700 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:19:53.127465 14700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: Bootstrap starting.
I20260925 13:19:53.130003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 14631
I20260925 13:19:53.130234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45323
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42013
--webserver_port=36453
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:19:53.135470 14700 log.cc:824] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:53.137326 14700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:19:53.137614 14700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: Bootstrap complete.
I20260925 13:19:53.139690 14700 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260925 13:19:53.139914 14700 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad83e0eda75740fbbed1bda0850e566d, State: Initialized, Role: FOLLOWER
I20260925 13:19:53.140053 14700 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260925 13:19:53.140112 14700 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:19:53.140156 14700 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:19:53.140202 14700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 1 FOLLOWER]: Advancing to term 2
I20260925 13:19:53.141152 14700 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260925 13:19:53.141239 14700 leader_election.cc:302] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad83e0eda75740fbbed1bda0850e566d; no voters:
I20260925 13:19:53.141390 14700 leader_election.cc:288] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 13:19:53.141443 14704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 13:19:53.141638 14704 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 2 LEADER]: Becoming Leader. State: Replica: ad83e0eda75740fbbed1bda0850e566d, State: Running, Role: LEADER
I20260925 13:19:53.141642 14700 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:19:53.141739 14704 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260925 13:19:53.141990 14705 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ad83e0eda75740fbbed1bda0850e566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } } }
I20260925 13:19:53.142002 14706 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad83e0eda75740fbbed1bda0850e566d. Latest consensus state: current_term: 2 leader_uuid: "ad83e0eda75740fbbed1bda0850e566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } } }
I20260925 13:19:53.142083 14705 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [sys.catalog]: This master's current role is: LEADER
I20260925 13:19:53.142091 14706 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [sys.catalog]: This master's current role is: LEADER
I20260925 13:19:53.142724 14710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:19:53.143498 14710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:19:53.143597 14710 catalog_manager.cc:1259] Loaded cluster ID: f281f850dc7441d386844db9e510c738
I20260925 13:19:53.143626 14710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:19:53.143844 14710 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 13:19:53.148480 14710 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 13:19:53.150928 14710 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:19:53.151153 14710 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: Loaded TSK: 0
I20260925 13:19:53.151389 14710 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 13:19:53.210054 14702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:53.210255 14702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:53.210285 14702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:53.211405 14702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:53.211445 14702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:53.211479 14702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:19:53.212877 14702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36453
--tserver_master_addrs=127.0.98.62:42013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:53.213109 14702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:53.213323 14702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:53.218106 14727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:53.218195 14728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:53.218343 14730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:53.218377 14702 server_base.cc:1034] running on GCE node
I20260925 13:19:53.218552 14702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:53.218782 14702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:53.219923 14702 hybrid_clock.cc:648] HybridClock initialized: now 1790342393219912 us; error 32 us; skew 500 ppm
Sep 25 13:19:53 dist-test-slave-t8d9 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:53.222256 14702 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:19:53.222648 14702 webserver.cc:460] Webserver started at http://127.0.98.1:36453/ using document root <none> and password file <none>
I20260925 13:19:53.222838 14702 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:53.222900 14702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:53.224267 14702 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:19:53.224934 14737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:53.225096 14702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:19:53.225174 14702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "9194a39b177a4b83a94683693cd20485"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "7f6a7310e4d4496a76d573d6697ae7e5"
server_key_iv: "41d12f7b55c79896bfba991157d42c5e"
server_key_version: "encryptionkey@0"
I20260925 13:19:53.225514 14702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:53.268694 14702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:53.270694 14702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:53.270807 14702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:53.271045 14702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:53.271504 14744 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 13:19:53.272930 14702 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 13:19:53.272974 14702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 13:19:53.273006 14702 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 13:19:53.273674 14702 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 13:19:53.273707 14702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260925 13:19:53.273765 14744 tablet_bootstrap.cc:492] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: Bootstrap starting.
I20260925 13:19:53.279364 14702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45323
I20260925 13:19:53.279451 14851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45323 every 8 connection(s)
I20260925 13:19:53.279772 14702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:19:53.283779 14744 log.cc:824] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:53.285107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 14702
I20260925 13:19:53.285362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45487
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42013
--webserver_port=37031
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:19:53.286228 14744 tablet_bootstrap.cc:492] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:19:53.286538 14744 tablet_bootstrap.cc:492] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: Bootstrap complete.
Sep 25 13:19:53 dist-test-slave-t8d9 krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:53.287606 14744 ts_tablet_manager.cc:1397] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.006s
I20260925 13:19:53.288780 14744 raft_consensus.cc:348] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:53.289104 14744 raft_consensus.cc:729] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9194a39b177a4b83a94683693cd20485, State: Initialized, Role: FOLLOWER
I20260925 13:19:53.289243 14744 consensus_queue.cc:260] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:53.289515 14744 ts_tablet_manager.cc:1428] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:19:53.291312 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.280583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42925 (local address 127.0.98.62:42013)
0925 13:19:53.280788 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:53.280800 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:53.280844 (+ 44us) server_negotiation.cc:407] Connection header received
0925 13:19:53.281481 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:53.281489 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:53.281593 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:53.281758 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:53.282369 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.283774 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.284971 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.285600 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.285719 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.287995 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:53.287999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:53.287999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:53.288011 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:53.289762 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:53.290178 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:53.290180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:53.290180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:53.290260 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:53.290487 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:53.290487 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:53.290488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:53.290680 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:53.290730 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:53.291066 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 13:19:53.291175 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260925 13:19:53.291396 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.280416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:42925)
0925 13:19:53.280682 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:53.280729 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:53.281212 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:53.281783 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:53.281787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:53.282044 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:53.282232 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.282237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.283906 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.283907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.284859 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.284862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.285728 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.285729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.285876 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.286459 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:53.286469 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:53.287896 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:53.289832 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:53.289838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:53.289840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:53.290107 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:53.290320 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:53.290320 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:53.290321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:53.290438 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:53.290750 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:53.290757 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:53.290912 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 13:19:53.291022 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 13:19:53.291201 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":75,"threads_started":1}
I20260925 13:19:53.292042 14852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42013
I20260925 13:19:53.292145 14852 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:53.292321 14852 heartbeater.cc:511] Master 127.0.98.62:42013 requested a full tablet report, sending...
I20260925 13:19:53.293033 14664 ts_manager.cc:194] Registered new tserver with Master: 9194a39b177a4b83a94683693cd20485 (127.0.98.1:45323)
I20260925 13:19:53.294040 14664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42925
I20260925 13:19:53.294807 14852 heartbeater.cc:503] Master 127.0.98.62:42013 was elected leader, sending a full tablet report...
Sep 25 13:19:53 dist-test-slave-t8d9 krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:19:53.302654 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.293618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45323 (local address 127.0.0.1:36434)
0925 13:19:53.293823 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:53.293854 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:53.293929 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:53.294196 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:53.294201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:53.294524 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:53.294581 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.294584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.295789 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.295790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.296337 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.296339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.296915 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.296917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.296962 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.297405 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:53.297408 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:53.298852 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:53.301435 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:53.301443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:53.301445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:53.301737 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:53.301963 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:53.301964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:53.301964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:53.302030 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:53.302331 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:53.302337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:53.302384 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 13:19:53.302449 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 13:19:53.302579 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
I20260925 13:19:53.302699 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.293670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.0.98.1:45323)
0925 13:19:53.293809 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:53.293814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:53.293922 (+ 108us) server_negotiation.cc:407] Connection header received
0925 13:19:53.294010 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:53.294012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:53.294027 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:53.294104 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:53.294673 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.295714 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.296415 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.296847 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.296921 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.298934 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:53.298942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:53.298942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:53.298950 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:53.301331 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:53.301812 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:53.301815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:53.301816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:53.301878 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:53.302096 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:53.302097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:53.302097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:53.302273 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:53.302348 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:53.302494 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 13:19:53.302602 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260925 13:19:53.303109 14786 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ac01a3a5644aa7adc7a7f22aa3481f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:19:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36434
I20260925 13:19:53.303702 14861 tablet_replica.cc:330] stopping tablet replica
I20260925 13:19:53.303798 14861 raft_consensus.cc:2229] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:19:53.303881 14861 raft_consensus.cc:2258] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:19:53.304169 14861 ts_tablet_manager.cc:1905] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:19:53.306255 14861 ts_tablet_manager.cc:1918] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:19:53.306325 14861 log.cc:1196] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal/wals/96ac01a3a5644aa7adc7a7f22aa3481f
I20260925 13:19:53.306556 14861 ts_tablet_manager.cc:1939] T 96ac01a3a5644aa7adc7a7f22aa3481f P 9194a39b177a4b83a94683693cd20485: Deleting consensus metadata
I20260925 13:19:53.307036 14649 catalog_manager.cc:4841] TS 9194a39b177a4b83a94683693cd20485 (127.0.98.1:45323): tablet 96ac01a3a5644aa7adc7a7f22aa3481f (table test-table [id=5f888484d07d480cb01cfa07bc2ddaad]) successfully deleted
W20260925 13:19:53.371347 14857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:53.371549 14857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:53.371572 14857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:53.372740 14857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:53.372779 14857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:53.372804 14857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:19:53.374191 14857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37031
--tserver_master_addrs=127.0.98.62:42013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:53.374392 14857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:53.374609 14857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:19:53.379374 14857 server_base.cc:1034] running on GCE node
W20260925 13:19:53.379317 14866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:53.379302 14869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:53.379293 14867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:53.379680 14857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:53.379894 14857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:53.381031 14857 hybrid_clock.cc:648] HybridClock initialized: now 1790342393381014 us; error 29 us; skew 500 ppm
Sep 25 13:19:53 dist-test-slave-t8d9 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:53.383423 14857 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:19:53.383855 14857 webserver.cc:460] Webserver started at http://127.0.98.2:37031/ using document root <none> and password file <none>
I20260925 13:19:53.384048 14857 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:53.384119 14857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:53.385447 14857 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:19:53.386212 14876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:53.386384 14857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:19:53.386474 14857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "d286e9d29072438da6a94fcb457836ef"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "28d01c50ff025189c816a11b8a576ceb"
server_key_iv: "e5babdaf2945742eb38b4ab3230dcab6"
server_key_version: "encryptionkey@0"
I20260925 13:19:53.386862 14857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:53.396870 14857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:53.399173 14857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:53.399330 14857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:53.399575 14857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:53.400022 14883 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 13:19:53.401408 14857 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 13:19:53.401471 14857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 13:19:53.401522 14857 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 13:19:53.402210 14857 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 13:19:53.402249 14857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 13:19:53.402305 14883 tablet_bootstrap.cc:492] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Bootstrap starting.
I20260925 13:19:53.403887 14883 log.cc:824] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:53.406790 14883 tablet_bootstrap.cc:492] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:19:53.407027 14883 tablet_bootstrap.cc:492] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Bootstrap complete.
I20260925 13:19:53.407169 14883 ts_tablet_manager.cc:1397] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 13:19:53.408473 14883 raft_consensus.cc:348] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:53.408710 14883 raft_consensus.cc:729] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d286e9d29072438da6a94fcb457836ef, State: Initialized, Role: FOLLOWER
I20260925 13:19:53.408859 14883 consensus_queue.cc:260] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:53.408951 14857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45487
I20260925 13:19:53.408999 14991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45487 every 8 connection(s)
I20260925 13:19:53.409152 14883 ts_tablet_manager.cc:1428] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 13:19:53.409312 14857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:19:53.412710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 14857
I20260925 13:19:53.412982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34749
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42013
--webserver_port=40831
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 13:19:53 dist-test-slave-t8d9 krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:53.418382 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.410013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54969 (local address 127.0.98.62:42013)
0925 13:19:53.410068 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:53.410070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:53.410256 (+ 186us) server_negotiation.cc:407] Connection header received
0925 13:19:53.410382 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:53.410383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:53.410394 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:53.410453 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:53.411034 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.411523 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.412523 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.412812 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.412879 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.415637 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:53.415639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:53.415640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:53.415648 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:53.416991 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:53.417433 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:53.417434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:53.417435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:53.417489 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:53.417756 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:53.417757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:53.417758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:53.417870 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:53.417927 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:53.418198 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 13:19:53.418289 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 13:19:53.418547 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.409974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:54969)
0925 13:19:53.410137 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:53.410177 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:53.410314 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:53.410503 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:53.410506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:53.410767 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:53.410958 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.410963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.411640 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.411641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.412422 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.412425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.412906 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.412907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.413029 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.413627 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:53.413635 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:53.415517 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:53.417075 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:53.417082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:53.417084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:53.417364 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:53.417565 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:53.417566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:53.417566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:53.417693 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:53.417929 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:53.417935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:53.418074 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 13:19:53.418168 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 13:19:53.418333 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260925 13:19:53.419003 14993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42013
I20260925 13:19:53.419126 14993 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:53.419319 14993 heartbeater.cc:511] Master 127.0.98.62:42013 requested a full tablet report, sending...
I20260925 13:19:53.419931 14664 ts_manager.cc:194] Registered new tserver with Master: d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487)
I20260925 13:19:53.420518 14664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54969
Sep 25 13:19:53 dist-test-slave-t8d9 krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260925 13:19:53.426980 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.420277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.0.98.2:45487)
0925 13:19:53.420418 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:53.420423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:53.420432 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:19:53.420462 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:53.420464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:53.420474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:53.420536 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:53.420853 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.421564 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.422225 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.422524 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.422579 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.424506 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:53.424514 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:53.424514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:53.424520 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:53.425948 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:53.426303 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:53.426305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:53.426306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:53.426353 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:53.426512 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:53.426513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:53.426513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:53.426637 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:53.426691 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:53.426835 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 13:19:53.426905 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260925 13:19:53.427008 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.420218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45487 (local address 127.0.0.1:56782)
0925 13:19:53.420260 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:53.420272 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:53.420320 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:53.420549 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:53.420550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:53.420700 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:53.420746 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.420748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.421659 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.421660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.422152 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.422153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.422589 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.422592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.422666 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.423179 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:53.423183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:53.424421 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:53.426021 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:53.426022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:53.426023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:53.426234 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:53.426407 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:53.426408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:53.426408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:53.426466 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:53.426693 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:53.426694 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:53.426737 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 13:19:53.426814 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 13:19:53.426909 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 13:19:53.427384 14926 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ac01a3a5644aa7adc7a7f22aa3481f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:19:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56782
I20260925 13:19:53.427985 14998 tablet_replica.cc:330] stopping tablet replica
I20260925 13:19:53.428083 14998 raft_consensus.cc:2229] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:19:53.428159 14998 raft_consensus.cc:2258] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:19:53.428436 14998 ts_tablet_manager.cc:1905] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:19:53.430630 14998 ts_tablet_manager.cc:1918] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:19:53.430703 14998 log.cc:1196] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/96ac01a3a5644aa7adc7a7f22aa3481f
I20260925 13:19:53.430886 14998 ts_tablet_manager.cc:1939] T 96ac01a3a5644aa7adc7a7f22aa3481f P d286e9d29072438da6a94fcb457836ef: Deleting consensus metadata
I20260925 13:19:53.431309 14649 catalog_manager.cc:4841] TS d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487): tablet 96ac01a3a5644aa7adc7a7f22aa3481f (table test-table [id=5f888484d07d480cb01cfa07bc2ddaad]) successfully deleted
W20260925 13:19:53.494707 14996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:53.494907 14996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:53.494930 14996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:53.496099 14996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:53.496140 14996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:53.496166 14996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:19:53.497635 14996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40831
--tserver_master_addrs=127.0.98.62:42013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:53.497857 14996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:53.498068 14996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:53.502833 15003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:53.502840 15006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:53.502840 15004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:53.503096 14996 server_base.cc:1034] running on GCE node
I20260925 13:19:53.503261 14996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:53.503471 14996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:53.504644 14996 hybrid_clock.cc:648] HybridClock initialized: now 1790342393504628 us; error 23 us; skew 500 ppm
Sep 25 13:19:53 dist-test-slave-t8d9 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:53.507316 14996 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:19:53.507712 14996 webserver.cc:460] Webserver started at http://127.0.98.3:40831/ using document root <none> and password file <none>
I20260925 13:19:53.507912 14996 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:53.507964 14996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:53.509253 14996 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:19:53.509793 15013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:53.509940 14996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:53.510022 14996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "c20564e854c642b5a313d613226ca35d"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "1a2cc5fb65a6e55c2755006a43e8829f"
server_key_iv: "dba6aa347acb17e04b8209fe720a1a24"
server_key_version: "encryptionkey@0"
I20260925 13:19:53.510329 14996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:53.520567 14996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:53.522370 14996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:53.522475 14996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:53.522644 14996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:53.522879 14996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:19:53.522908 14996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:53.522927 14996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:19:53.522943 14996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:53.528185 14996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34749
I20260925 13:19:53.528258 15126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34749 every 8 connection(s)
I20260925 13:19:53.528527 14996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:19:53 dist-test-slave-t8d9 krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:19:53.536907 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.529196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40825 (local address 127.0.98.62:42013)
0925 13:19:53.529240 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:53.529242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:53.529466 (+ 224us) server_negotiation.cc:407] Connection header received
0925 13:19:53.529986 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:53.529988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:53.530000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:53.530052 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:53.530948 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.531469 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.532429 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.532615 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.532670 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.534601 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:53.534603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:53.534604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:53.534612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:53.535652 (+ 1040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:53.536026 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:53.536027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:53.536028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:53.536079 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:53.536304 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:53.536305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:53.536306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:53.536413 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:53.536468 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:53.536730 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 13:19:53.536815 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:19:53.537043 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.529194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:40825)
0925 13:19:53.529351 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:53.529382 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:53.529889 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:53.530133 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:53.530137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:53.530667 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:53.530852 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.530857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.531552 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.531553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.532347 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.532349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.532686 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.532687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.532809 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.533243 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:53.533251 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:53.534512 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:53.535744 (+ 1232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:53.535751 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:53.535753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:53.535960 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:53.536137 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:53.536137 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:53.536138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:53.536245 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:53.536468 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:53.536475 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:53.536605 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 13:19:53.536691 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 13:19:53.536837 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260925 13:19:53.537565 15127 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42013
I20260925 13:19:53.537658 15127 heartbeater.cc:464] Registering TS with master...
I20260925 13:19:53.537824 15127 heartbeater.cc:511] Master 127.0.98.62:42013 requested a full tablet report, sending...
I20260925 13:19:53.538193 14664 ts_manager.cc:194] Registered new tserver with Master: c20564e854c642b5a313d613226ca35d (127.0.98.3:34749)
I20260925 13:19:53.538246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 14996
I20260925 13:19:53.538585 14664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40825
I20260925 13:19:53.543768 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.538591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.0.98.62:42013)
0925 13:19:53.538629 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:53.538631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:53.538775 (+ 144us) server_negotiation.cc:407] Connection header received
0925 13:19:53.538817 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:53.538818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:53.538826 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:53.538863 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:53.539367 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.539799 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.540481 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.540665 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.540721 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.541829 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:53.541830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:53.541830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:53.541836 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:53.543001 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:53.543288 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:53.543288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:53.543289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:53.543327 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:53.543430 (+ 103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:53.543431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:53.543431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:53.543509 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:53.543620 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:53.543654 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 13:19:53.543705 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:19:53.544452 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:19:53.547158 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.544778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.0.98.62:42013)
0925 13:19:53.544818 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:53.544820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:53.544890 (+ 70us) server_negotiation.cc:407] Connection header received
0925 13:19:53.544956 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:53.544957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:53.544988 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:53.545049 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:19:53.545139 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.545828 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.546615 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.546785 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.546834 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.546945 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:19:53.547028 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:19:53.547086 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:53.547102 (+ 16us) server_negotiation.cc:299] Negotiation successful
0925 13:19:53.547114 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:19:53.547379 14664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:19:53.548877 14664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:19:53.550746 14786 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7e2e3471754f1ea9ecdccbbc6c8ba4 (DEFAULT_TABLE table=test-table [id=4a3190568e494d81972a52fcc0d84696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:19:53.550746 14926 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7e2e3471754f1ea9ecdccbbc6c8ba4 (DEFAULT_TABLE table=test-table [id=4a3190568e494d81972a52fcc0d84696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:19:53.550920 14926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7e2e3471754f1ea9ecdccbbc6c8ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:53.550920 14786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7e2e3471754f1ea9ecdccbbc6c8ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:53.552410 14883 tablet_bootstrap.cc:492] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef: Bootstrap starting.
I20260925 13:19:53.552791 14883 tablet_bootstrap.cc:654] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:53.553097 14744 tablet_bootstrap.cc:492] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485: Bootstrap starting.
I20260925 13:19:53.553370 14744 tablet_bootstrap.cc:654] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:53.553503 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.550483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34749 (local address 127.0.0.1:44842)
0925 13:19:53.550524 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:53.550534 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:53.550569 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:53.550914 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:53.550915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:53.550916 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:53.550951 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.550952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.551794 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.551795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.552546 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.552548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.553273 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.553274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.553333 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.553350 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 13:19:53.553401 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 13:19:53.553441 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 13:19:53.553521 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.550583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.0.98.3:34749)
0925 13:19:53.550755 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:53.550759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:53.550768 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:19:53.550800 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:53.550801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:53.550813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:53.551016 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:53.551082 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.551704 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.552615 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.553203 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.553260 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.553316 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:53.553415 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 13:19:53.553448 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20260925 13:19:53.553823 14883 tablet_bootstrap.cc:492] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef: No bootstrap required, opened a new log
I20260925 13:19:53.553876 14883 ts_tablet_manager.cc:1397] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 13:19:53.554018 14883 raft_consensus.cc:348] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:53.554080 14883 raft_consensus.cc:374] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:53.554105 14883 raft_consensus.cc:729] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d286e9d29072438da6a94fcb457836ef, State: Initialized, Role: FOLLOWER
I20260925 13:19:53.554150 14883 consensus_queue.cc:260] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:53.554241 14883 ts_tablet_manager.cc:1428] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:53.554159 15061 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7e2e3471754f1ea9ecdccbbc6c8ba4 (DEFAULT_TABLE table=test-table [id=4a3190568e494d81972a52fcc0d84696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:19:53.554286 14993 heartbeater.cc:503] Master 127.0.98.62:42013 was elected leader, sending a full tablet report...
I20260925 13:19:53.554433 14744 tablet_bootstrap.cc:492] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485: No bootstrap required, opened a new log
I20260925 13:19:53.554481 14744 ts_tablet_manager.cc:1397] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 13:19:53.554478 15061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7e2e3471754f1ea9ecdccbbc6c8ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:19:53.554793 14744 raft_consensus.cc:348] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:53.554863 14744 raft_consensus.cc:374] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:53.554903 14744 raft_consensus.cc:729] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9194a39b177a4b83a94683693cd20485, State: Initialized, Role: FOLLOWER
I20260925 13:19:53.554942 14744 consensus_queue.cc:260] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:53.555020 14744 ts_tablet_manager.cc:1428] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 13:19:53.557245 15135 tablet_bootstrap.cc:492] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d: Bootstrap starting.
I20260925 13:19:53.557850 15135 tablet_bootstrap.cc:654] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d: Neither blocks nor log segments found. Creating new log.
I20260925 13:19:53.558240 15135 log.cc:824] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:53.558848 15135 tablet_bootstrap.cc:492] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d: No bootstrap required, opened a new log
I20260925 13:19:53.558899 15135 ts_tablet_manager.cc:1397] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 13:19:53.560276 15135 raft_consensus.cc:348] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:53.560406 15135 raft_consensus.cc:374] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:19:53.560426 15135 raft_consensus.cc:729] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20564e854c642b5a313d613226ca35d, State: Initialized, Role: FOLLOWER
I20260925 13:19:53.560506 15135 consensus_queue.cc:260] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:53.560709 15135 ts_tablet_manager.cc:1428] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 13:19:53.560775 15127 heartbeater.cc:503] Master 127.0.98.62:42013 was elected leader, sending a full tablet report...
W20260925 13:19:53.659955 14994 tablet.cc:2369] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:19:53.763861 14858 raft_consensus.cc:482] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:19:53.764113 14858 raft_consensus.cc:504] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:53.764484 14858 leader_election.cc:288] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c20564e854c642b5a313d613226ca35d (127.0.98.3:34749), d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487)
I20260925 13:19:53.767460 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.764760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33083 (local address 127.0.98.3:34749)
0925 13:19:53.764800 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:53.764802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:53.764812 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:19:53.764902 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:53.764903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:53.764915 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:53.764964 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:53.765146 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.765727 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.766706 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.767090 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.767141 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.767150 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:53.767352 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 13:19:53.767391 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:19:53.767473 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.764682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34749 (local address 127.0.98.1:33083)
0925 13:19:53.764740 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:53.764752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:53.764805 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:53.764985 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:53.764987 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:53.764989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:53.765073 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.765075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.765803 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.765804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.766612 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.766615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.767161 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.767162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.767247 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.767276 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 13:19:53.767339 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 13:19:53.767381 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 13:19:53.767633 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.764841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46681 (local address 127.0.98.2:45487)
0925 13:19:53.764910 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:53.764912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:53.764924 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:19:53.765021 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:53.765022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:53.765051 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:53.765136 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:19:53.765294 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.765851 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.766761 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.767270 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.767346 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.767423 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:53.767513 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 13:19:53.767555 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 13:19:53.767802 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.764711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45487 (local address 127.0.98.1:46681)
0925 13:19:53.764870 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:53.764879 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:53.764919 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:53.765144 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:53.765145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:53.765147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:19:53.765212 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.765213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.765938 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.765939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.766676 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:53.766678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.767371 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.767372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:53.767447 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.767451 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:19:53.767627 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 13:19:53.767662 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":80,"threads_started":1}
I20260925 13:19:53.767887 15081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7e2e3471754f1ea9ecdccbbc6c8ba4" candidate_uuid: "9194a39b177a4b83a94683693cd20485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20564e854c642b5a313d613226ca35d" is_pre_election: true
I20260925 13:19:53.768051 15081 raft_consensus.cc:2454] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9194a39b177a4b83a94683693cd20485 in term 0.
I20260925 13:19:53.768127 14946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7e2e3471754f1ea9ecdccbbc6c8ba4" candidate_uuid: "9194a39b177a4b83a94683693cd20485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d286e9d29072438da6a94fcb457836ef" is_pre_election: true
I20260925 13:19:53.768275 14946 raft_consensus.cc:2454] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9194a39b177a4b83a94683693cd20485 in term 0.
I20260925 13:19:53.768327 14738 leader_election.cc:302] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9194a39b177a4b83a94683693cd20485, c20564e854c642b5a313d613226ca35d; no voters:
I20260925 13:19:53.768455 14858 raft_consensus.cc:2781] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:19:53.768501 14858 raft_consensus.cc:482] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:19:53.768523 14858 raft_consensus.cc:3037] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:53.769187 14858 raft_consensus.cc:504] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:53.769317 14858 leader_election.cc:288] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 1 election: Requested vote from peers c20564e854c642b5a313d613226ca35d (127.0.98.3:34749), d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487)
I20260925 13:19:53.769510 15081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7e2e3471754f1ea9ecdccbbc6c8ba4" candidate_uuid: "9194a39b177a4b83a94683693cd20485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20564e854c642b5a313d613226ca35d"
I20260925 13:19:53.769518 14946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7e2e3471754f1ea9ecdccbbc6c8ba4" candidate_uuid: "9194a39b177a4b83a94683693cd20485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d286e9d29072438da6a94fcb457836ef"
I20260925 13:19:53.769621 15081 raft_consensus.cc:3037] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:53.769634 14946 raft_consensus.cc:3037] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:19:53.770292 15081 raft_consensus.cc:2454] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9194a39b177a4b83a94683693cd20485 in term 1.
I20260925 13:19:53.770360 14946 raft_consensus.cc:2454] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9194a39b177a4b83a94683693cd20485 in term 1.
I20260925 13:19:53.770519 14738 leader_election.cc:302] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9194a39b177a4b83a94683693cd20485, c20564e854c642b5a313d613226ca35d; no voters:
I20260925 13:19:53.770632 14858 raft_consensus.cc:2781] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:19:53.770783 14858 raft_consensus.cc:686] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 LEADER]: Becoming Leader. State: Replica: 9194a39b177a4b83a94683693cd20485, State: Running, Role: LEADER
I20260925 13:19:53.770846 14858 consensus_queue.cc:237] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:53.771566 14663 catalog_manager.cc:5495] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9194a39b177a4b83a94683693cd20485 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9194a39b177a4b83a94683693cd20485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } health_report { overall_health: UNKNOWN } } }
W20260925 13:19:53.779033 15128 tablet.cc:2369] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:19:53.780573 14853 tablet.cc:2369] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:19:53.783807 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.781596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36446 (local address 127.0.98.1:45323)
0925 13:19:53.781651 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:53.781653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:53.781662 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:19:53.781733 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:53.781734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:53.781753 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:53.781812 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:19:53.781924 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.782410 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.783108 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:53.783308 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:53.783361 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:53.783501 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:19:53.783603 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:19:53.783662 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:53.783713 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 13:19:53.783737 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:19:53.785089 15081 raft_consensus.cc:1261] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d [term 1 FOLLOWER]: Refusing update from remote peer 9194a39b177a4b83a94683693cd20485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:19:53.785112 14946 raft_consensus.cc:1261] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef [term 1 FOLLOWER]: Refusing update from remote peer 9194a39b177a4b83a94683693cd20485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:19:53.785439 15139 consensus_queue.cc:1035] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:19:53.785597 14858 consensus_queue.cc:1035] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:19:53.787299 15141 mvcc.cc:205] Tried to move back new op lower bound from 7333242444941746176 to 7333242444886061056. Current Snapshot: MvccSnapshot[applied={T|T < 7333242444941746176}]
I20260925 13:19:53.787592 15143 mvcc.cc:205] Tried to move back new op lower bound from 7333242444941746176 to 7333242444886061056. Current Snapshot: MvccSnapshot[applied={T|T < 7333242444941746176}]
I20260925 13:19:53.788323 15142 mvcc.cc:205] Tried to move back new op lower bound from 7333242444941746176 to 7333242444886061056. Current Snapshot: MvccSnapshot[applied={T|T < 7333242444941746176 or (T in {7333242444941746176})}]
I20260925 13:19:53.789505 14663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:19:53.789613 14663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:19:53.790266 14663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: Sending DeleteTablet for 3 replicas of tablet 4a7e2e3471754f1ea9ecdccbbc6c8ba4
I20260925 13:19:53.790540 14786 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a7e2e3471754f1ea9ecdccbbc6c8ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:19:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36434
I20260925 13:19:53.790581 14926 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a7e2e3471754f1ea9ecdccbbc6c8ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:19:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56782
I20260925 13:19:53.790561 15061 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a7e2e3471754f1ea9ecdccbbc6c8ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:19:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44842
I20260925 13:19:53.790655 14998 tablet_replica.cc:330] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef: stopping tablet replica
I20260925 13:19:53.790704 14998 raft_consensus.cc:2229] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:19:53.790748 14998 raft_consensus.cc:2258] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:19:53.790745 14861 tablet_replica.cc:330] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485: stopping tablet replica
I20260925 13:19:53.790807 14861 raft_consensus.cc:2229] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:19:53.790877 14861 raft_consensus.cc:2258] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:19:53.791064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 14702
I20260925 13:19:53.791154 14861 ts_tablet_manager.cc:1905] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:19:53.791149 14998 ts_tablet_manager.cc:1905] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:19:53.791673 15150 tablet_replica.cc:330] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d: stopping tablet replica
I20260925 13:19:53.791770 15150 raft_consensus.cc:2229] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:19:53.791867 15150 raft_consensus.cc:2258] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:19:53.792285 15150 ts_tablet_manager.cc:1905] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:19:53.792657 14998 ts_tablet_manager.cc:1918] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:19:53.792699 14998 log.cc:1196] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/4a7e2e3471754f1ea9ecdccbbc6c8ba4
I20260925 13:19:53.792830 14998 ts_tablet_manager.cc:1939] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P d286e9d29072438da6a94fcb457836ef: Deleting consensus metadata
I20260925 13:19:53.793421 14649 catalog_manager.cc:4841] TS d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487): tablet 4a7e2e3471754f1ea9ecdccbbc6c8ba4 (table test-table [id=4a3190568e494d81972a52fcc0d84696]) successfully deleted
I20260925 13:19:53.794159 15150 ts_tablet_manager.cc:1918] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:19:53.794222 15150 log.cc:1196] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/4a7e2e3471754f1ea9ecdccbbc6c8ba4
I20260925 13:19:53.794446 15150 ts_tablet_manager.cc:1939] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P c20564e854c642b5a313d613226ca35d: Deleting consensus metadata
I20260925 13:19:53.794857 14648 catalog_manager.cc:4841] TS c20564e854c642b5a313d613226ca35d (127.0.98.3:34749): tablet 4a7e2e3471754f1ea9ecdccbbc6c8ba4 (table test-table [id=4a3190568e494d81972a52fcc0d84696]) successfully deleted
W20260925 13:19:53.796386 14649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45323 (error 108)
I20260925 13:19:53.796660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 14857
I20260925 13:19:53.797210 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:53.796747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45323 (local address 127.0.0.1:36456)
0925 13:19:53.797126 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:53.797177 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358}
W20260925 13:19:53.797350 14649 catalog_manager.cc:4584] TS 9194a39b177a4b83a94683693cd20485 (127.0.98.1:45323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a7e2e3471754f1ea9ecdccbbc6c8ba4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45323: connect: Connection refused (error 111)
I20260925 13:19:53.801244 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 14996
I20260925 13:19:53.805473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 14631
I20260925 13:19:53.809795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42013
--webserver_interface=127.0.98.62
--webserver_port=38837
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:19:53.888864 15152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:53.889074 15152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:53.889103 15152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:53.890292 15152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:19:53.890336 15152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:53.890354 15152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:53.890369 15152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:19:53.890384 15152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:19:53.891909 15152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:53.892172 15152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:53.892385 15152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:53.897024 15160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:53.897070 15157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:53.897073 15152 server_base.cc:1034] running on GCE node
W20260925 13:19:53.897010 15158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:53.897509 15152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:53.897812 15152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:53.898970 15152 hybrid_clock.cc:648] HybridClock initialized: now 1790342393898957 us; error 28 us; skew 500 ppm
Sep 25 13:19:53 dist-test-slave-t8d9 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:53.901374 15152 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:19:53.901834 15152 webserver.cc:460] Webserver started at http://127.0.98.62:38837/ using document root <none> and password file <none>
I20260925 13:19:53.902052 15152 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:53.902100 15152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:53.903376 15152 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:19:53.904014 15167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:53.904192 15152 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:19:53.904275 15152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "ad83e0eda75740fbbed1bda0850e566d"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "ec32d488ae11ad703efaa59836f5facd"
server_key_iv: "4a1d2a5fe0550130ca31c61da651761a"
server_key_version: "encryptionkey@0"
I20260925 13:19:53.904616 15152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:53.914434 15152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:53.916272 15152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:53.916409 15152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:53.920501 15152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42013
I20260925 13:19:53.920547 15219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42013 every 8 connection(s)
I20260925 13:19:53.920890 15152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:19:53.922437 15220 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:19:53.923097 15220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: Bootstrap starting.
I20260925 13:19:53.924069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 15152
I20260925 13:19:53.924317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45323
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42013
--webserver_port=36453
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:19:53.931411 15220 log.cc:824] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:53.934011 15220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:19:53.934340 15220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d: Bootstrap complete.
I20260925 13:19:53.936452 15220 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260925 13:19:53.936674 15220 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad83e0eda75740fbbed1bda0850e566d, State: Initialized, Role: FOLLOWER
I20260925 13:19:53.936827 15220 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260925 13:19:53.936901 15220 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:19:53.936949 15220 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:19:53.936993 15220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 2 FOLLOWER]: Advancing to term 3
I20260925 13:19:53.938004 15220 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260925 13:19:53.938102 15220 leader_election.cc:302] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad83e0eda75740fbbed1bda0850e566d; no voters:
I20260925 13:19:53.938257 15220 leader_election.cc:288] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 13:19:53.938304 15224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 3 FOLLOWER]: Leader election won for term 3
I20260925 13:19:53.938414 15224 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [term 3 LEADER]: Becoming Leader. State: Replica: ad83e0eda75740fbbed1bda0850e566d, State: Running, Role: LEADER
I20260925 13:19:53.938519 15220 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:19:53.938494 15224 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260925 13:19:53.938858 15226 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad83e0eda75740fbbed1bda0850e566d. Latest consensus state: current_term: 3 leader_uuid: "ad83e0eda75740fbbed1bda0850e566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } } }
I20260925 13:19:53.939004 15226 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [sys.catalog]: This master's current role is: LEADER
I20260925 13:19:53.939179 15225 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ad83e0eda75740fbbed1bda0850e566d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad83e0eda75740fbbed1bda0850e566d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } } }
I20260925 13:19:53.939280 15225 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad83e0eda75740fbbed1bda0850e566d [sys.catalog]: This master's current role is: LEADER
I20260925 13:19:53.939385 15233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:19:53.940129 15233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:19:53.940217 15233 catalog_manager.cc:1259] Loaded cluster ID: f281f850dc7441d386844db9e510c738
I20260925 13:19:53.940320 15233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 13:19:53.940712 15233 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fec7aada9ad google::LogMessage::Fail() at ??:0
@ 0x7fec7aadd88a google::LogMessage::SendToLog() at ??:0
@ 0x7fec7aada4de google::LogMessage::Flush() at ??:0
@ 0x7fec7aadba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fec829fb0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fec82a048b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fec7c9a80d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fec7c99d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fec7d8016db start_thread at ??:0
@ 0x7fec795d861f clone at ??:0
W20260925 13:19:54.015452 15222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:54.015663 15222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:54.015688 15222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:54.016884 15222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:54.016925 15222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:54.016978 15222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:19:54.018574 15222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36453
--tserver_master_addrs=127.0.98.62:42013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:54.018813 15222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:54.019040 15222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:54.023959 15256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:54.024106 15222 server_base.cc:1034] running on GCE node
W20260925 13:19:54.023940 15259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:54.023963 15257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:54.024399 15222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:54.024653 15222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:54.025830 15222 hybrid_clock.cc:648] HybridClock initialized: now 1790342394025812 us; error 32 us; skew 500 ppm
Sep 25 13:19:54 dist-test-slave-t8d9 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:54.028649 15222 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:19:54.029239 15222 webserver.cc:460] Webserver started at http://127.0.98.1:36453/ using document root <none> and password file <none>
I20260925 13:19:54.029503 15222 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:54.029580 15222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:54.031340 15222 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:19:54.032276 15266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:54.032532 15222 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:19:54.032629 15222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "9194a39b177a4b83a94683693cd20485"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "7f6a7310e4d4496a76d573d6697ae7e5"
server_key_iv: "41d12f7b55c79896bfba991157d42c5e"
server_key_version: "encryptionkey@0"
I20260925 13:19:54.032984 15222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:54.046396 15222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:54.048446 15222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:54.048629 15222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:54.048844 15222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:54.049255 15273 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 13:19:54.051167 15222 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 13:19:54.051211 15222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 13:19:54.051244 15222 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 13:19:54.051842 15222 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 13:19:54.051873 15222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 13:19:54.051971 15273 tablet_bootstrap.cc:492] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485: Bootstrap starting.
I20260925 13:19:54.053757 15273 log.cc:824] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485: Log is configured to *not* fsync() on all Append() calls
I20260925 13:19:54.056660 15273 tablet_bootstrap.cc:492] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:19:54.056941 15273 tablet_bootstrap.cc:492] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485: Bootstrap complete.
I20260925 13:19:54.057107 15273 ts_tablet_manager.cc:1397] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 13:19:54.058352 15222 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45323
I20260925 13:19:54.058441 15381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45323 every 8 connection(s)
I20260925 13:19:54.058703 15222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:19:54.058969 15273 raft_consensus.cc:348] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:54.059336 15273 raft_consensus.cc:729] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9194a39b177a4b83a94683693cd20485, State: Initialized, Role: FOLLOWER
I20260925 13:19:54.059496 15273 consensus_queue.cc:260] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:54.059624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 15222
I20260925 13:19:54.059757 15273 ts_tablet_manager.cc:1428] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20260925 13:19:54.059862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45487
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42013
--webserver_port=37031
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:19:54.150701 15384 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:19:54.059304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:53673)
0925 13:19:54.059467 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:54.059508 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:54.059678 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:54.150489 (+ 90811us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
W20260925 13:19:54.151058 15382 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42013 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42013: Client connection negotiation failed: client connection to 127.0.98.62:42013: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 13:19:54.151428 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:54.151287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:40463)
0925 13:19:54.151345 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:54.151397 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 13:19:54.151878 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:54.151770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:51865)
0925 13:19:54.151820 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:54.151851 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 13:19:54.152048 15382 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 13:19:54.158017 15386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:54.158234 15386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:54.158269 15386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:54.159466 15386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:54.159508 15386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:54.159552 15386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:19:54.161029 15386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37031
--tserver_master_addrs=127.0.98.62:42013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:54.161278 15386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:54.161499 15386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:54.166431 15394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:54.166481 15392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:54.166472 15391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:54.166715 15386 server_base.cc:1034] running on GCE node
I20260925 13:19:54.166926 15386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:54.167177 15386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:54.168336 15386 hybrid_clock.cc:648] HybridClock initialized: now 1790342394168321 us; error 37 us; skew 500 ppm
Sep 25 13:19:54 dist-test-slave-t8d9 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:54.170861 15386 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:19:54.171311 15386 webserver.cc:460] Webserver started at http://127.0.98.2:37031/ using document root <none> and password file <none>
I20260925 13:19:54.171541 15386 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:54.171597 15386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:54.172926 15386 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:19:54.173581 15401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:54.173751 15386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:19:54.173837 15386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "d286e9d29072438da6a94fcb457836ef"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "28d01c50ff025189c816a11b8a576ceb"
server_key_iv: "e5babdaf2945742eb38b4ab3230dcab6"
server_key_version: "encryptionkey@0"
I20260925 13:19:54.174185 15386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:54.184980 15386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:54.187038 15386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:54.187182 15386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:54.187405 15386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:54.187706 15386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:19:54.187738 15386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:54.187767 15386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:19:54.187785 15386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:54.193387 15386 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45487
I20260925 13:19:54.193459 15514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45487 every 8 connection(s)
I20260925 13:19:54.193820 15386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:19:54.194826 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:54.194461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:47059)
0925 13:19:54.194613 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:54.194685 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
W20260925 13:19:54.195232 15515 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42013 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42013: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
I20260925 13:19:54.195261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 15386
I20260925 13:19:54.195513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34749
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42013
--webserver_port=40831
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:19:54.195506 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:54.195420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:35553)
0925 13:19:54.195461 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:54.195487 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 13:19:54.195943 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:54.195877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:48619)
0925 13:19:54.195909 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:54.195929 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 13:19:54.196043 15515 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 13:19:54.275375 15518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:19:54.275590 15518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:19:54.275614 15518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:19:54.276801 15518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:19:54.276846 15518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:19:54.276880 15518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:19:54.278398 15518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40831
--tserver_master_addrs=127.0.98.62:42013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:19:54.278605 15518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:19:54.278793 15518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:19:54.283919 15526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:54.283979 15518 server_base.cc:1034] running on GCE node
W20260925 13:19:54.283929 15523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:19:54.284205 15524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:19:54.284435 15518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:19:54.284680 15518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:19:54.285818 15518 hybrid_clock.cc:648] HybridClock initialized: now 1790342394285807 us; error 28 us; skew 500 ppm
Sep 25 13:19:54 dist-test-slave-t8d9 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:19:54.288271 15518 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:19:54.288692 15518 webserver.cc:460] Webserver started at http://127.0.98.3:40831/ using document root <none> and password file <none>
I20260925 13:19:54.288887 15518 fs_manager.cc:359] Metadata directory not provided
I20260925 13:19:54.288930 15518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:19:54.290334 15518 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:19:54.290894 15533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:54.291057 15518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:19:54.291128 15518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "c20564e854c642b5a313d613226ca35d"
format_stamp: "Formatted at 2026-09-25 13:19:52 on dist-test-slave-t8d9"
server_key: "1a2cc5fb65a6e55c2755006a43e8829f"
server_key_iv: "dba6aa347acb17e04b8209fe720a1a24"
server_key_version: "encryptionkey@0"
I20260925 13:19:54.291491 15518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:19:54.312263 15518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:19:54.314538 15518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:19:54.314661 15518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:19:54.314919 15518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:19:54.315235 15518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:19:54.315271 15518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:54.315294 15518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:19:54.315304 15518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:19:54.320853 15518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34749
I20260925 13:19:54.320906 15646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34749 every 8 connection(s)
I20260925 13:19:54.321244 15518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:19:54.322310 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:54.321944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:42107)
0925 13:19:54.322087 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:54.322171 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
W20260925 13:19:54.322701 15647 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42013 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42013: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
I20260925 13:19:54.322949 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:54.322868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:60833)
0925 13:19:54.322905 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:54.322932 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 13:19:54.323201 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:54.323132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:46695)
0925 13:19:54.323166 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:54.323187 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 13:19:54.323312 15647 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 13:19:54.330579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 15518
I20260925 13:19:55.152727 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:55.152441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:56075)
0925 13:19:55.152617 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:55.152663 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
I20260925 13:19:55.196696 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:55.196421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:33525)
0925 13:19:55.196568 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:55.196638 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":60,"threads_started":1}
I20260925 13:19:55.323939 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:55.323674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:39855)
0925 13:19:55.323829 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:55.323872 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260925 13:19:55.389096 15654 raft_consensus.cc:482] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:19:55.389236 15654 raft_consensus.cc:504] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:55.389601 15654 leader_election.cc:288] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c20564e854c642b5a313d613226ca35d (127.0.98.3:34749), d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487)
Sep 25 13:19:55 dist-test-slave-t8d9 krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 25 13:19:55 dist-test-slave-t8d9 krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260925 13:19:55.409961 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:55.389765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45487 (local address 127.0.98.1:41575)
0925 13:19:55.389890 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:55.389900 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:55.389951 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:55.390801 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:55.390804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:55.391195 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:55.391630 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:55.391633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:55.392697 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:55.392698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:55.393478 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:55.393480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:55.394073 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:55.394079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:55.394202 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:55.403706 (+ 9504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:55.403712 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:55.405556 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:55.408272 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:55.408273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:55.408273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:55.408477 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:55.408730 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:55.408731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:55.408731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:55.408786 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:55.409217 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:55.409226 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:55.409417 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 13:19:55.409608 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 13:19:55.409816 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":59,"thread_start_us":62,"threads_started":1}
I20260925 13:19:55.409994 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:55.389868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37895 (local address 127.0.98.3:34749)
0925 13:19:55.390030 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:55.390043 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:55.390076 (+ 33us) server_negotiation.cc:407] Connection header received
0925 13:19:55.390561 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:55.390563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:55.390602 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:55.390741 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:55.391595 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:55.392509 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:55.393572 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:55.393984 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:55.394138 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:55.405968 (+ 11830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:55.405974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:55.405974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:55.405989 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:55.407783 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:55.408294 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:55.408299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:55.408299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:55.408427 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:55.408745 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:55.408746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:55.408746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:55.409121 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:55.409202 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:55.409720 (+ 518us) server_negotiation.cc:299] Negotiation successful
0925 13:19:55.409868 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":53,"threads_started":1}
I20260925 13:19:55.410001 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:55.389914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41575 (local address 127.0.98.2:45487)
0925 13:19:55.390047 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 13:19:55.390057 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:19:55.390089 (+ 32us) server_negotiation.cc:407] Connection header received
0925 13:19:55.390543 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:19:55.390545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:19:55.390592 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:19:55.390729 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:19:55.391742 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:55.392595 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:55.393572 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:19:55.393983 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:19:55.394103 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:55.405727 (+ 11624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:19:55.405733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:19:55.405733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:19:55.405748 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:19:55.407783 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:55.408568 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:55.408572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:55.408572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:55.408660 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:19:55.408843 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:19:55.408844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:19:55.408845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:19:55.409112 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:19:55.409202 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 13:19:55.409720 (+ 518us) server_negotiation.cc:299] Negotiation successful
0925 13:19:55.409869 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":54,"threads_started":1}
I20260925 13:19:55.410020 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:55.389748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34749 (local address 127.0.98.1:37895)
0925 13:19:55.389790 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:55.389801 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:19:55.389864 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:19:55.390801 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:19:55.390804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:19:55.391195 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:19:55.391436 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:55.391442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:55.392646 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:55.392647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:55.393475 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:19:55.393477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:19:55.394117 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:19:55.394118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:19:55.394205 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:19:55.403690 (+ 9485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:19:55.403703 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:19:55.405822 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:19:55.407894 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:55.407903 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:55.407905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:55.408179 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:55.408511 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:19:55.408512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:19:55.408512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:19:55.408666 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:19:55.409217 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:19:55.409226 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:19:55.409423 (+ 197us) client_negotiation.cc:769] Sending connection context
0925 13:19:55.409643 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 13:19:55.409884 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":163}
W20260925 13:19:55.411299 15268 leader_election.cc:341] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487): Not found: Tablet not found: 4a7e2e3471754f1ea9ecdccbbc6c8ba4
W20260925 13:19:55.411448 15267 leader_election.cc:341] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c20564e854c642b5a313d613226ca35d (127.0.98.3:34749): Not found: Tablet not found: 4a7e2e3471754f1ea9ecdccbbc6c8ba4
I20260925 13:19:55.411520 15267 leader_election.cc:302] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9194a39b177a4b83a94683693cd20485; no voters: c20564e854c642b5a313d613226ca35d, d286e9d29072438da6a94fcb457836ef
I20260925 13:19:55.411639 15654 raft_consensus.cc:2726] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 13:19:56.153492 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:56.153241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:48355)
0925 13:19:56.153397 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:56.153442 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260925 13:19:56.197448 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:56.197209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:52791)
0925 13:19:56.197360 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:56.197402 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260925 13:19:56.324630 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:56.324398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:52279)
0925 13:19:56.324542 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:56.324584 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
I20260925 13:19:56.992911 15661 raft_consensus.cc:482] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:19:56.993010 15661 raft_consensus.cc:504] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:56.993144 15661 leader_election.cc:288] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c20564e854c642b5a313d613226ca35d (127.0.98.3:34749), d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487)
W20260925 13:19:56.993512 15267 leader_election.cc:341] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c20564e854c642b5a313d613226ca35d (127.0.98.3:34749): Not found: Tablet not found: 4a7e2e3471754f1ea9ecdccbbc6c8ba4
W20260925 13:19:56.993642 15268 leader_election.cc:341] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487): Not found: Tablet not found: 4a7e2e3471754f1ea9ecdccbbc6c8ba4
I20260925 13:19:56.993706 15268 leader_election.cc:302] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9194a39b177a4b83a94683693cd20485; no voters: c20564e854c642b5a313d613226ca35d, d286e9d29072438da6a94fcb457836ef
I20260925 13:19:56.993765 15661 raft_consensus.cc:2726] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 13:19:57.154255 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:57.153988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:57741)
0925 13:19:57.154147 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:57.154193 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260925 13:19:57.198096 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:57.197880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:36731)
0925 13:19:57.198008 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:57.198047 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260925 13:19:57.325337 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:57.325089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:34145)
0925 13:19:57.325224 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:57.325294 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260925 13:19:58.154927 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:58.154702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:33851)
0925 13:19:58.154837 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:58.154879 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260925 13:19:58.198734 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:58.198497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:55235)
0925 13:19:58.198642 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:58.198688 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":60,"threads_started":1}
I20260925 13:19:58.326066 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:58.325822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:44523)
0925 13:19:58.325967 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:58.326010 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260925 13:19:59.155714 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:59.155464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:39765)
0925 13:19:59.155618 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:59.155666 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":60,"threads_started":1}
I20260925 13:19:59.195130 15669 raft_consensus.cc:482] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:19:59.195218 15669 raft_consensus.cc:504] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:19:59.195380 15669 leader_election.cc:288] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c20564e854c642b5a313d613226ca35d (127.0.98.3:34749), d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487)
W20260925 13:19:59.195725 15268 leader_election.cc:341] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487): Not found: Tablet not found: 4a7e2e3471754f1ea9ecdccbbc6c8ba4
W20260925 13:19:59.195820 15267 leader_election.cc:341] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c20564e854c642b5a313d613226ca35d (127.0.98.3:34749): Not found: Tablet not found: 4a7e2e3471754f1ea9ecdccbbc6c8ba4
I20260925 13:19:59.195871 15267 leader_election.cc:302] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9194a39b177a4b83a94683693cd20485; no voters: c20564e854c642b5a313d613226ca35d, d286e9d29072438da6a94fcb457836ef
I20260925 13:19:59.195942 15669 raft_consensus.cc:2726] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 13:19:59.199360 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:59.199175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:55527)
0925 13:19:59.199283 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:59.199320 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":65,"threads_started":1}
I20260925 13:19:59.326721 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:19:59.326520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:60655)
0925 13:19:59.326638 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 13:19:59.326674 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260925 13:20:00.156337 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:00.156095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:57995)
0925 13:20:00.156245 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:00.156288 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260925 13:20:00.199965 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:00.199770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:42859)
0925 13:20:00.199887 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:00.199923 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260925 13:20:00.327406 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:00.327196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:36107)
0925 13:20:00.327315 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:00.327358 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260925 13:20:01.157166 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:01.156903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:51803)
0925 13:20:01.157060 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:01.157117 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260925 13:20:01.200659 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:01.200441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:43913)
0925 13:20:01.200574 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:01.200607 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260925 13:20:01.328177 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:01.327930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:36287)
0925 13:20:01.328079 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:01.328128 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260925 13:20:02.157966 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:02.157745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:43943)
0925 13:20:02.157874 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:02.157919 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260925 13:20:02.201395 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:02.201140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:47965)
0925 13:20:02.201299 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:02.201345 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260925 13:20:02.329000 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:02.328761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:52265)
0925 13:20:02.328900 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:02.328949 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260925 13:20:02.487699 15681 raft_consensus.cc:482] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:20:02.487803 15681 raft_consensus.cc:504] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:20:02.487972 15681 leader_election.cc:288] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c20564e854c642b5a313d613226ca35d (127.0.98.3:34749), d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487)
W20260925 13:20:02.488415 15268 leader_election.cc:341] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487): Not found: Tablet not found: 4a7e2e3471754f1ea9ecdccbbc6c8ba4
W20260925 13:20:02.488520 15267 leader_election.cc:341] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c20564e854c642b5a313d613226ca35d (127.0.98.3:34749): Not found: Tablet not found: 4a7e2e3471754f1ea9ecdccbbc6c8ba4
I20260925 13:20:02.488576 15267 leader_election.cc:302] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9194a39b177a4b83a94683693cd20485; no voters: c20564e854c642b5a313d613226ca35d, d286e9d29072438da6a94fcb457836ef
I20260925 13:20:02.488649 15681 raft_consensus.cc:2726] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 13:20:03.158756 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:03.158432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:55721)
0925 13:20:03.158650 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:03.158703 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":71,"threads_started":1}
I20260925 13:20:03.202216 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:03.201961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:45471)
0925 13:20:03.202115 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:03.202165 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260925 13:20:03.329815 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:03.329522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:55449)
0925 13:20:03.329721 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:03.329768 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":108,"threads_started":1}
I20260925 13:20:04.159503 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:04.159271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:38477)
0925 13:20:04.159406 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:04.159455 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260925 13:20:04.202960 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:04.202732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:37279)
0925 13:20:04.202870 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:04.202912 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260925 13:20:04.330513 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:04.330275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:44799)
0925 13:20:04.330425 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:04.330466 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260925 13:20:05.160197 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:05.159992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:42615)
0925 13:20:05.160112 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:05.160152 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260925 13:20:05.203682 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:05.203451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:40507)
0925 13:20:05.203594 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:05.203635 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260925 13:20:05.331212 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:05.330971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:45083)
0925 13:20:05.331124 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:05.331165 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
I20260925 13:20:06.161003 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:06.160739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:52915)
0925 13:20:06.160904 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:06.160954 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":71,"threads_started":1}
I20260925 13:20:06.204524 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:06.204264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:34799)
0925 13:20:06.204427 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:06.204476 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":68,"threads_started":1}
I20260925 13:20:06.315982 15693 raft_consensus.cc:482] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:20:06.316092 15693 raft_consensus.cc:504] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:20:06.316273 15693 leader_election.cc:288] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c20564e854c642b5a313d613226ca35d (127.0.98.3:34749), d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487)
W20260925 13:20:06.316764 15268 leader_election.cc:341] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487): Not found: Tablet not found: 4a7e2e3471754f1ea9ecdccbbc6c8ba4
W20260925 13:20:06.316856 15267 leader_election.cc:341] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c20564e854c642b5a313d613226ca35d (127.0.98.3:34749): Not found: Tablet not found: 4a7e2e3471754f1ea9ecdccbbc6c8ba4
I20260925 13:20:06.316893 15267 leader_election.cc:302] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9194a39b177a4b83a94683693cd20485; no voters: c20564e854c642b5a313d613226ca35d, d286e9d29072438da6a94fcb457836ef
I20260925 13:20:06.316972 15693 raft_consensus.cc:2726] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 13:20:06.332078 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:06.331856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:55703)
0925 13:20:06.331991 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:06.332029 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
I20260925 13:20:07.161684 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:07.161452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:45241)
0925 13:20:07.161594 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:07.161634 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260925 13:20:07.205312 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:07.205040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:48425)
0925 13:20:07.205197 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:07.205246 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260925 13:20:07.332753 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:07.332514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:57767)
0925 13:20:07.332655 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:07.332704 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260925 13:20:08.162379 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:08.162166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:53453)
0925 13:20:08.162289 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:08.162334 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260925 13:20:08.206072 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:08.205846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:56285)
0925 13:20:08.205982 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:08.206026 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260925 13:20:08.263497 15700 raft_consensus.cc:482] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:20:08.263625 15700 raft_consensus.cc:504] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20564e854c642b5a313d613226ca35d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "9194a39b177a4b83a94683693cd20485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45323 } } peers { permanent_uuid: "d286e9d29072438da6a94fcb457836ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } }
I20260925 13:20:08.263813 15700 leader_election.cc:288] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c20564e854c642b5a313d613226ca35d (127.0.98.3:34749), d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487)
W20260925 13:20:08.264163 15267 leader_election.cc:341] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c20564e854c642b5a313d613226ca35d (127.0.98.3:34749): Not found: Tablet not found: 4a7e2e3471754f1ea9ecdccbbc6c8ba4
W20260925 13:20:08.264254 15268 leader_election.cc:341] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d286e9d29072438da6a94fcb457836ef (127.0.98.2:45487): Not found: Tablet not found: 4a7e2e3471754f1ea9ecdccbbc6c8ba4
I20260925 13:20:08.264321 15268 leader_election.cc:302] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9194a39b177a4b83a94683693cd20485; no voters: c20564e854c642b5a313d613226ca35d, d286e9d29072438da6a94fcb457836ef
I20260925 13:20:08.264405 15700 raft_consensus.cc:2726] T 4a7e2e3471754f1ea9ecdccbbc6c8ba4 P 9194a39b177a4b83a94683693cd20485 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 13:20:08.333482 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:08.333228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:40673)
0925 13:20:08.333387 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:08.333434 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260925 13:20:09.163203 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:09.162954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:52525)
0925 13:20:09.163116 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:09.163156 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":64,"threads_started":1}
I20260925 13:20:09.206743 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:09.206536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:36181)
0925 13:20:09.206651 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:09.206690 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260925 13:20:09.334236 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:09.334018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:39409)
0925 13:20:09.334147 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:09.334189 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260925 13:20:09.337570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 15222
I20260925 13:20:09.341670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 15386
I20260925 13:20:09.345263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 15518
2026-09-25T13:20:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18346 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:20:09 dist-test-slave-t8d9 krb5kdc[15707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:20:09 dist-test-slave-t8d9 krb5kdc[15707](info): set up 2 sockets
Sep 25 13:20:09 dist-test-slave-t8d9 krb5kdc[15707](info): commencing operation
krb5kdc: starting...
W20260925 13:20:10.313370 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:20:10 dist-test-slave-t8d9 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:20:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:20:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:20:10.432629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34629
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:20:10.513036 15720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:20:10.513257 15720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:20:10.513279 15720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:20:10.514464 15720 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 13:20:10.514503 15720 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 13:20:10.514515 15720 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 13:20:10.514525 15720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:20:10.514537 15720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:20:10.514549 15720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:20:10.514559 15720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:20:10.514571 15720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:20:10.516080 15720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:20:10.516326 15720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:20:10.516582 15720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:20:10.521421 15728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:10.521451 15725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:10.521569 15720 server_base.cc:1034] running on GCE node
W20260925 13:20:10.521438 15726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:10.521953 15720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:20:10.522265 15720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:20:10.523415 15720 hybrid_clock.cc:648] HybridClock initialized: now 1790342410523397 us; error 35 us; skew 500 ppm
Sep 25 13:20:10 dist-test-slave-t8d9 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:20:10.525879 15720 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:20:10.526332 15720 webserver.cc:460] Webserver started at http://127.0.98.62:41691/ using document root <none> and password file <none>
I20260925 13:20:10.526547 15720 fs_manager.cc:359] Metadata directory not provided
I20260925 13:20:10.526597 15720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:20:10.526717 15720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:20:10.527771 15720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98"
format_stamp: "Formatted at 2026-09-25 13:20:10 on dist-test-slave-t8d9"
server_key: "3bb6fae1afde7ac07739a147bc59969b"
server_key_iv: "2aa63f660f2166242e0bbff4a9e85456"
server_key_version: "encryptionkey@0"
I20260925 13:20:10.528096 15720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98"
format_stamp: "Formatted at 2026-09-25 13:20:10 on dist-test-slave-t8d9"
server_key: "3bb6fae1afde7ac07739a147bc59969b"
server_key_iv: "2aa63f660f2166242e0bbff4a9e85456"
server_key_version: "encryptionkey@0"
I20260925 13:20:10.529371 15720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:20:10.530166 15735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:10.530339 15720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:10.530421 15720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98"
format_stamp: "Formatted at 2026-09-25 13:20:10 on dist-test-slave-t8d9"
server_key: "3bb6fae1afde7ac07739a147bc59969b"
server_key_iv: "2aa63f660f2166242e0bbff4a9e85456"
server_key_version: "encryptionkey@0"
I20260925 13:20:10.530490 15720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:20:10.544287 15720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:20:10.546350 15720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:20:10.546485 15720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:20:10.550163 15720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34629
I20260925 13:20:10.550199 15787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34629 every 8 connection(s)
I20260925 13:20:10.550529 15720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:20:10.551369 15788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:20:10.553617 15788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Bootstrap starting.
I20260925 13:20:10.554134 15788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Neither blocks nor log segments found. Creating new log.
I20260925 13:20:10.554364 15788 log.cc:824] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Log is configured to *not* fsync() on all Append() calls
I20260925 13:20:10.554991 15788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: No bootstrap required, opened a new log
I20260925 13:20:10.556486 15788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260925 13:20:10.556658 15788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:20:10.556690 15788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3dd3a2fd304aa282576ffa8bd2fc98, State: Initialized, Role: FOLLOWER
I20260925 13:20:10.556797 15788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260925 13:20:10.556849 15788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:20:10.556869 15788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:20:10.556897 15788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:20:10.557353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 15720
I20260925 13:20:10.557448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:20:10.557451 15788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260925 13:20:10.557579 15788 leader_election.cc:302] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e3dd3a2fd304aa282576ffa8bd2fc98; no voters:
I20260925 13:20:10.557667 392 external_mini_cluster.cc:1428] Setting key 119cd0cb85f450ea5d138b6d9673bcb1
I20260925 13:20:10.557772 15788 leader_election.cc:288] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:20:10.557803 15791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:20:10.557950 15791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 1 LEADER]: Becoming Leader. State: Replica: 9e3dd3a2fd304aa282576ffa8bd2fc98, State: Running, Role: LEADER
I20260925 13:20:10.558023 15788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:20:10.558037 15791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260925 13:20:10.558650 15793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } } }
I20260925 13:20:10.558748 15793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [sys.catalog]: This master's current role is: LEADER
I20260925 13:20:10.558647 15792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e3dd3a2fd304aa282576ffa8bd2fc98. Latest consensus state: current_term: 1 leader_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } } }
I20260925 13:20:10.558950 15792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [sys.catalog]: This master's current role is: LEADER
I20260925 13:20:10.559055 15802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:20:10.559665 15802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:20:10.562254 15802 catalog_manager.cc:1347] Generated new cluster ID: 0717d2a1aebe495a82fc228ff874c19c
I20260925 13:20:10.562301 15802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:20:10 dist-test-slave-t8d9 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:20:10.569499 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:10.558331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49494 (local address 127.0.98.62:34629)
0925 13:20:10.558711 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:10.558723 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:10.558759 (+ 36us) server_negotiation.cc:407] Connection header received
0925 13:20:10.558882 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:10.558890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:10.558947 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:10.559892 (+ 945us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:20:10.559985 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:10.561349 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:10.562221 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:10.562857 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:10.562980 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:10.565797 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:20:10.565803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:20:10.565804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:20:10.565818 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:20:10.567819 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:10.568244 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:10.568247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:10.568248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:10.568353 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:10.568546 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:10.568546 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:10.568547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:10.568864 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:20:10.568992 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:10.569129 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 13:20:10.569274 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":67,"threads_started":1}
I20260925 13:20:10.590178 15802 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:20:10.590760 15802 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:20:10.594882 15802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Generated new TSK 0
I20260925 13:20:10.595062 15802 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:20:10.626215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34629
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:20:10.707268 15814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:20:10.707486 15814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:20:10.707510 15814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:20:10.708668 15814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:20:10.708716 15814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:20:10.708750 15814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:20:10.710227 15814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:20:10.710459 15814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:20:10.710692 15814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:20:10.715490 15819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:10.715521 15820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:10.715523 15814 server_base.cc:1034] running on GCE node
W20260925 13:20:10.715503 15822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:10.715823 15814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:20:10.716032 15814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:20:10.717170 15814 hybrid_clock.cc:648] HybridClock initialized: now 1790342410717156 us; error 30 us; skew 500 ppm
Sep 25 13:20:10 dist-test-slave-t8d9 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:20:10.719632 15814 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:20:10.720067 15814 webserver.cc:460] Webserver started at http://127.0.98.1:37167/ using document root <none> and password file <none>
I20260925 13:20:10.720293 15814 fs_manager.cc:359] Metadata directory not provided
I20260925 13:20:10.720343 15814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:20:10.720469 15814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:20:10.721571 15814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "6f8f41766d1e44a38206e3c63e115c93"
format_stamp: "Formatted at 2026-09-25 13:20:10 on dist-test-slave-t8d9"
server_key: "a43ae166631e50146a51f545b68f508a"
server_key_iv: "f1cb3f2cffaf0e666a0a20b271fb7cea"
server_key_version: "encryptionkey@0"
I20260925 13:20:10.721928 15814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6f8f41766d1e44a38206e3c63e115c93"
format_stamp: "Formatted at 2026-09-25 13:20:10 on dist-test-slave-t8d9"
server_key: "a43ae166631e50146a51f545b68f508a"
server_key_iv: "f1cb3f2cffaf0e666a0a20b271fb7cea"
server_key_version: "encryptionkey@0"
I20260925 13:20:10.723182 15814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:20:10.723861 15829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:10.723992 15814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:20:10.724074 15814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "6f8f41766d1e44a38206e3c63e115c93"
format_stamp: "Formatted at 2026-09-25 13:20:10 on dist-test-slave-t8d9"
server_key: "a43ae166631e50146a51f545b68f508a"
server_key_iv: "f1cb3f2cffaf0e666a0a20b271fb7cea"
server_key_version: "encryptionkey@0"
I20260925 13:20:10.724140 15814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:20:10.748440 15814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:20:10.750608 15814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:20:10.750757 15814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:20:10.751001 15814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:20:10.751346 15814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:20:10.751387 15814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:10.751418 15814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:20:10.751439 15814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:10.756984 15814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46107
I20260925 13:20:10.757037 15942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46107 every 8 connection(s)
I20260925 13:20:10.757367 15814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:20:10.761631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 15814
I20260925 13:20:10.761750 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:20:10.761958 392 external_mini_cluster.cc:1428] Setting key 8e10cb4c49347a3e407bdf6f9ca57aa0
Sep 25 13:20:10 dist-test-slave-t8d9 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:20:10.767166 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:10.758112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58773 (local address 127.0.98.62:34629)
0925 13:20:10.758163 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:10.758164 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:10.758371 (+ 207us) server_negotiation.cc:407] Connection header received
0925 13:20:10.758914 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:10.758916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:10.758933 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:10.758972 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:20:10.759958 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:10.760457 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:10.761451 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:10.761692 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:10.761761 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:10.764289 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:20:10.764291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:20:10.764292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:20:10.764301 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:20:10.765775 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:10.766157 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:10.766158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:10.766159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:10.766234 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:10.766492 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:10.766493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:10.766493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:10.766639 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:20:10.766712 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:10.766965 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 13:20:10.767064 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:20:10.767243 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:10.758091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:58773)
0925 13:20:10.758252 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:10.758290 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:10.758812 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:10.759049 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:10.759054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:10.759643 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:20:10.759854 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:10.759859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:10.760543 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:10.760544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:10.761364 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:10.761366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:10.761790 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:10.761791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:10.761957 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:10.762495 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:20:10.762506 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:20:10.764180 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:20:10.765851 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:10.765858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:10.765860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:10.766088 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:10.766318 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:10.766319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:10.766319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:10.766432 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:10.766699 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:20:10.766702 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:20:10.766838 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 13:20:10.766920 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 13:20:10.767058 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260925 13:20:10.767794 15943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34629
I20260925 13:20:10.767886 15943 heartbeater.cc:464] Registering TS with master...
I20260925 13:20:10.768069 15943 heartbeater.cc:511] Master 127.0.98.62:34629 requested a full tablet report, sending...
I20260925 13:20:10.768594 15752 ts_manager.cc:194] Registered new tserver with Master: 6f8f41766d1e44a38206e3c63e115c93 (127.0.98.1:46107)
I20260925 13:20:10.769162 15752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58773
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:20:10.787875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34629
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:20:10.869586 15948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:20:10.869819 15948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:20:10.869844 15948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:20:10.871035 15948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:20:10.871083 15948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:20:10.871110 15948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:20:10.872557 15948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:20:10.872802 15948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:20:10.873023 15948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:20:10.877875 15954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:10.877897 15956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:10.878013 15948 server_base.cc:1034] running on GCE node
W20260925 13:20:10.878119 15953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:10.878325 15948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:20:10.878580 15948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:20:10.879772 15948 hybrid_clock.cc:648] HybridClock initialized: now 1790342410879753 us; error 31 us; skew 500 ppm
Sep 25 13:20:10 dist-test-slave-t8d9 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:20:10.882460 15948 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:20:10.882949 15948 webserver.cc:460] Webserver started at http://127.0.98.2:36289/ using document root <none> and password file <none>
I20260925 13:20:10.883164 15948 fs_manager.cc:359] Metadata directory not provided
I20260925 13:20:10.883217 15948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:20:10.883349 15948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:20:10.884410 15948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "171c50c01da9426585e416bc06cb627a"
format_stamp: "Formatted at 2026-09-25 13:20:10 on dist-test-slave-t8d9"
server_key: "4ef07e36c66ad90fc3a8fc0bf81d8a41"
server_key_iv: "7d41831090be7bb11bfc987653fc4cd2"
server_key_version: "encryptionkey@0"
I20260925 13:20:10.884743 15948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "171c50c01da9426585e416bc06cb627a"
format_stamp: "Formatted at 2026-09-25 13:20:10 on dist-test-slave-t8d9"
server_key: "4ef07e36c66ad90fc3a8fc0bf81d8a41"
server_key_iv: "7d41831090be7bb11bfc987653fc4cd2"
server_key_version: "encryptionkey@0"
I20260925 13:20:10.886094 15948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:20:10.886794 15963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:10.886945 15948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:20:10.887028 15948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "171c50c01da9426585e416bc06cb627a"
format_stamp: "Formatted at 2026-09-25 13:20:10 on dist-test-slave-t8d9"
server_key: "4ef07e36c66ad90fc3a8fc0bf81d8a41"
server_key_iv: "7d41831090be7bb11bfc987653fc4cd2"
server_key_version: "encryptionkey@0"
I20260925 13:20:10.887096 15948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:20:10.900187 15948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:20:10.902259 15948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:20:10.902403 15948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:20:10.902628 15948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:20:10.902927 15948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:20:10.902961 15948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:10.902989 15948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:20:10.903014 15948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:10.908572 15948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42729
I20260925 13:20:10.908623 16076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42729 every 8 connection(s)
I20260925 13:20:10.908939 15948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:20:10.912763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 15948
I20260925 13:20:10.912885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:20:10.913086 392 external_mini_cluster.cc:1428] Setting key 64da541cec40f325e982d621d237a06b
Sep 25 13:20:10 dist-test-slave-t8d9 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:20:10.918314 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:10.909639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59357 (local address 127.0.98.62:34629)
0925 13:20:10.909688 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:10.909690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:10.909910 (+ 220us) server_negotiation.cc:407] Connection header received
0925 13:20:10.910393 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:10.910394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:10.910406 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:10.910456 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:20:10.911387 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:10.911887 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:10.913024 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:10.913254 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:10.913312 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:10.915645 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:20:10.915646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:20:10.915647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:20:10.915656 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:20:10.916959 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:10.917342 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:10.917343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:10.917344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:10.917404 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:10.917686 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:10.917687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:10.917687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:10.917808 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:20:10.917864 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:10.918135 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 13:20:10.918225 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:20:10.918489 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:10.909646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:59357)
0925 13:20:10.909792 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:10.909829 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:10.910306 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:10.910520 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:10.910524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:10.911093 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:20:10.911299 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:10.911304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:10.911970 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:10.911971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:10.912944 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:10.912947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:10.913343 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:10.913345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:10.913465 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:10.913988 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:20:10.913998 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:20:10.915550 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:20:10.917037 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:10.917045 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:10.917048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:10.917277 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:10.917458 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:10.917459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:10.917459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:10.917623 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:10.917861 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:20:10.917868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:20:10.918010 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 13:20:10.918095 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 13:20:10.918269 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260925 13:20:10.919051 16077 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34629
I20260925 13:20:10.919147 16077 heartbeater.cc:464] Registering TS with master...
I20260925 13:20:10.919332 16077 heartbeater.cc:511] Master 127.0.98.62:34629 requested a full tablet report, sending...
I20260925 13:20:10.919695 15752 ts_manager.cc:194] Registered new tserver with Master: 171c50c01da9426585e416bc06cb627a (127.0.98.2:42729)
I20260925 13:20:10.920047 15752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59357
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:20:10.936385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34629
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:20:11.016230 16082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:20:11.016471 16082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:20:11.016497 16082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:20:11.017773 16082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:20:11.017824 16082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:20:11.017856 16082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:20:11.019372 16082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:20:11.019608 16082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:20:11.019865 16082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:20:11.024715 16090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:11.024762 16088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:11.024695 16087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:11.024875 16082 server_base.cc:1034] running on GCE node
I20260925 13:20:11.025103 16082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:20:11.025357 16082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:20:11.026525 16082 hybrid_clock.cc:648] HybridClock initialized: now 1790342411026488 us; error 45 us; skew 500 ppm
Sep 25 13:20:11 dist-test-slave-t8d9 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:20:11.029212 16082 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:20:11.029700 16082 webserver.cc:460] Webserver started at http://127.0.98.3:35183/ using document root <none> and password file <none>
I20260925 13:20:11.029912 16082 fs_manager.cc:359] Metadata directory not provided
I20260925 13:20:11.029963 16082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:20:11.030093 16082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:20:11.031205 16082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "a395365b68ce4658ae97b8d4d8e20e98"
format_stamp: "Formatted at 2026-09-25 13:20:11 on dist-test-slave-t8d9"
server_key: "0abed15cd72af8e6f74a5a4690512628"
server_key_iv: "e8f3c42313d82e27f7f476d53ecb44ce"
server_key_version: "encryptionkey@0"
I20260925 13:20:11.031545 16082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a395365b68ce4658ae97b8d4d8e20e98"
format_stamp: "Formatted at 2026-09-25 13:20:11 on dist-test-slave-t8d9"
server_key: "0abed15cd72af8e6f74a5a4690512628"
server_key_iv: "e8f3c42313d82e27f7f476d53ecb44ce"
server_key_version: "encryptionkey@0"
I20260925 13:20:11.032826 16082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:20:11.033507 16097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:11.033684 16082 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:20:11.033772 16082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "a395365b68ce4658ae97b8d4d8e20e98"
format_stamp: "Formatted at 2026-09-25 13:20:11 on dist-test-slave-t8d9"
server_key: "0abed15cd72af8e6f74a5a4690512628"
server_key_iv: "e8f3c42313d82e27f7f476d53ecb44ce"
server_key_version: "encryptionkey@0"
I20260925 13:20:11.033840 16082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:20:11.050457 16082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:20:11.052714 16082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:20:11.052836 16082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:20:11.053037 16082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:20:11.053305 16082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:20:11.053336 16082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:11.053359 16082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:20:11.053369 16082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:11.058959 16082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42191
I20260925 13:20:11.059024 16210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42191 every 8 connection(s)
I20260925 13:20:11.059348 16082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:20:11.061021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 16082
I20260925 13:20:11.061116 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:20:11.061309 392 external_mini_cluster.cc:1428] Setting key 2094fb76fd00d2ccdd60706cba7b0c02
Sep 25 13:20:11 dist-test-slave-t8d9 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:20:11.068591 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:11.060091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47459 (local address 127.0.98.62:34629)
0925 13:20:11.060140 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:11.060142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:11.060367 (+ 225us) server_negotiation.cc:407] Connection header received
0925 13:20:11.060896 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:11.060897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:11.060911 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:11.060963 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:20:11.062104 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.062642 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.063681 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.063874 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.063920 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:11.066112 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:20:11.066114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:20:11.066114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:20:11.066121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:20:11.067242 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:11.067689 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:11.067690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:11.067690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:11.067754 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:11.068041 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:11.068042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:11.068043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:11.068160 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:20:11.068212 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:11.068450 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 13:20:11.068523 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:20:11.068799 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:11.060088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:47459)
0925 13:20:11.060242 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:11.060280 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:11.060789 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:11.061041 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:11.061045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:11.061765 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:20:11.062010 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:11.062019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.062732 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.062733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:11.063620 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:11.063623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.063945 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.063946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:11.064051 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:11.064570 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:20:11.064579 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:20:11.066008 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:20:11.067339 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:11.067347 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:11.067349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:11.067623 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:11.067830 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:11.067831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:11.067831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:11.067976 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:11.068210 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:20:11.068214 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:20:11.068344 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 13:20:11.068433 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 13:20:11.068573 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260925 13:20:11.069271 16211 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34629
I20260925 13:20:11.069368 16211 heartbeater.cc:464] Registering TS with master...
I20260925 13:20:11.069577 16211 heartbeater.cc:511] Master 127.0.98.62:34629 requested a full tablet report, sending...
I20260925 13:20:11.069989 15752 ts_manager.cc:194] Registered new tserver with Master: a395365b68ce4658ae97b8d4d8e20e98 (127.0.98.3:42191)
I20260925 13:20:11.070348 15752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47459
I20260925 13:20:11.072881 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:20:11.080147 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:11.074257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49502 (local address 127.0.98.62:34629)
0925 13:20:11.074305 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:11.074307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:11.074345 (+ 38us) server_negotiation.cc:407] Connection header received
0925 13:20:11.074424 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:11.074424 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:11.074436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:11.074476 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:20:11.075001 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.075495 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.076202 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.076391 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.076445 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:11.077815 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:20:11.077817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:20:11.077817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:20:11.077826 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:20:11.078977 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:11.079322 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:11.079323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:11.079323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:11.079377 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:11.079545 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:11.079546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:11.079546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:11.079747 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:20:11.079841 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:11.079967 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 13:20:11.080052 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:20:11.081974 15752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:20:11.083678 15752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:20:11.089723 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:11.085821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42729 (local address 127.0.0.1:46364)
0925 13:20:11.086034 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:11.086080 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:11.086150 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:11.086376 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:11.086379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:11.086381 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:20:11.086458 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:11.086460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.087436 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.087437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:11.088359 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:11.088361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.089360 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.089361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:11.089458 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:11.089473 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 13:20:11.089571 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 13:20:11.089632 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":92,"threads_started":1}
I20260925 13:20:11.089766 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:11.085948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.0.98.2:42729)
0925 13:20:11.086146 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:11.086152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:11.086163 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:20:11.086280 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:11.086282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:11.086298 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:11.086366 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:20:11.086521 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.087343 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.088436 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.089269 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.089337 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:11.089397 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:11.089600 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 13:20:11.089643 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
I20260925 13:20:11.090106 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:11.085822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46107 (local address 127.0.0.1:58858)
0925 13:20:11.086200 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:11.086209 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:11.086250 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:11.086530 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:11.086531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:11.086534 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:20:11.086582 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:11.086583 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.087561 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.087562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:11.088853 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:11.088856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.089880 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.089881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:11.089950 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:11.089954 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:20:11.090001 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 13:20:11.090041 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260925 13:20:11.090337 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:11.086012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58858 (local address 127.0.98.1:46107)
0925 13:20:11.086398 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:11.086402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:11.086410 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:20:11.086440 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:11.086442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:11.086453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:11.086519 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:20:11.086679 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.087442 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.088941 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.089711 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.089782 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:11.089843 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:11.090217 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 13:20:11.090263 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":85,"threads_started":1}
I20260925 13:20:11.090476 16011 tablet_service.cc:1467] Processing CreateTablet for tablet 3198795f27784167bbd9b4995e5188d2 (DEFAULT_TABLE table=test-table [id=b9b7ef762cc6430ebe2560b2b2d5a79e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:20:11.090838 16011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3198795f27784167bbd9b4995e5188d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:20:11.090960 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:11.086058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42191 (local address 127.0.0.1:40798)
0925 13:20:11.086233 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:11.086243 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:11.086294 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:11.087784 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:11.087785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:11.087788 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:20:11.087824 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:11.087825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.088975 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.088976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:11.089888 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:11.089891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.090731 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.090732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:11.090797 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:11.090800 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:20:11.090853 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 13:20:11.090891 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":49,"threads_started":1}
I20260925 13:20:11.090975 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:11.086216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40798 (local address 127.0.98.3:42191)
0925 13:20:11.087629 (+ 1413us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:11.087633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:11.087642 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:20:11.087677 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:11.087679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:11.087694 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:11.087770 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:20:11.087889 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.088901 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.089970 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.090663 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.090729 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:11.090786 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:11.090867 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 13:20:11.090900 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1362,"thread_start_us":66,"threads_started":1}
I20260925 13:20:11.091738 16145 tablet_service.cc:1467] Processing CreateTablet for tablet 3198795f27784167bbd9b4995e5188d2 (DEFAULT_TABLE table=test-table [id=b9b7ef762cc6430ebe2560b2b2d5a79e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:20:11.092094 16145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3198795f27784167bbd9b4995e5188d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:20:11.094461 15877 tablet_service.cc:1467] Processing CreateTablet for tablet 3198795f27784167bbd9b4995e5188d2 (DEFAULT_TABLE table=test-table [id=b9b7ef762cc6430ebe2560b2b2d5a79e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:20:11.094882 15877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3198795f27784167bbd9b4995e5188d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:20:11.095582 16225 tablet_bootstrap.cc:492] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98: Bootstrap starting.
I20260925 13:20:11.095806 16226 tablet_bootstrap.cc:492] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a: Bootstrap starting.
I20260925 13:20:11.096364 16226 tablet_bootstrap.cc:654] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a: Neither blocks nor log segments found. Creating new log.
I20260925 13:20:11.096381 16225 tablet_bootstrap.cc:654] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98: Neither blocks nor log segments found. Creating new log.
I20260925 13:20:11.096827 16226 log.cc:824] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a: Log is configured to *not* fsync() on all Append() calls
I20260925 13:20:11.096877 16225 log.cc:824] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98: Log is configured to *not* fsync() on all Append() calls
I20260925 13:20:11.097523 16226 tablet_bootstrap.cc:492] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a: No bootstrap required, opened a new log
I20260925 13:20:11.097620 16226 ts_tablet_manager.cc:1397] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:20:11.097642 16225 tablet_bootstrap.cc:492] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98: No bootstrap required, opened a new log
I20260925 13:20:11.097705 16225 ts_tablet_manager.cc:1397] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:20:11.098328 16229 tablet_bootstrap.cc:492] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: Bootstrap starting.
I20260925 13:20:11.099062 16229 tablet_bootstrap.cc:654] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: Neither blocks nor log segments found. Creating new log.
I20260925 13:20:11.099424 16225 raft_consensus.cc:348] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } }
I20260925 13:20:11.099576 16225 raft_consensus.cc:374] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:20:11.099610 16225 raft_consensus.cc:729] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a395365b68ce4658ae97b8d4d8e20e98, State: Initialized, Role: FOLLOWER
I20260925 13:20:11.099646 16229 log.cc:824] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: Log is configured to *not* fsync() on all Append() calls
I20260925 13:20:11.099637 16226 raft_consensus.cc:348] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } }
I20260925 13:20:11.099768 16225 consensus_queue.cc:260] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } }
I20260925 13:20:11.099829 16226 raft_consensus.cc:374] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:20:11.099875 16226 raft_consensus.cc:729] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c50c01da9426585e416bc06cb627a, State: Initialized, Role: FOLLOWER
I20260925 13:20:11.099985 16226 consensus_queue.cc:260] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } }
I20260925 13:20:11.100042 16225 ts_tablet_manager.cc:1428] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:20:11.100082 16211 heartbeater.cc:503] Master 127.0.98.62:34629 was elected leader, sending a full tablet report...
I20260925 13:20:11.100250 16226 ts_tablet_manager.cc:1428] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 13:20:11.100294 16077 heartbeater.cc:503] Master 127.0.98.62:34629 was elected leader, sending a full tablet report...
I20260925 13:20:11.100701 16229 tablet_bootstrap.cc:492] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: No bootstrap required, opened a new log
I20260925 13:20:11.100776 16229 ts_tablet_manager.cc:1397] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 13:20:11.102381 16229 raft_consensus.cc:348] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } }
I20260925 13:20:11.102522 16229 raft_consensus.cc:374] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:20:11.102543 16229 raft_consensus.cc:729] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8f41766d1e44a38206e3c63e115c93, State: Initialized, Role: FOLLOWER
I20260925 13:20:11.102633 16229 consensus_queue.cc:260] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } }
I20260925 13:20:11.102895 16229 ts_tablet_manager.cc:1428] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 13:20:11.102907 15943 heartbeater.cc:503] Master 127.0.98.62:34629 was elected leader, sending a full tablet report...
I20260925 13:20:11.103962 16231 raft_consensus.cc:482] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:20:11.104051 16231 raft_consensus.cc:504] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } }
I20260925 13:20:11.104357 16231 leader_election.cc:288] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c50c01da9426585e416bc06cb627a (127.0.98.2:42729), 6f8f41766d1e44a38206e3c63e115c93 (127.0.98.1:46107)
I20260925 13:20:11.107144 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:11.104635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50421 (local address 127.0.98.1:46107)
0925 13:20:11.104677 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:11.104679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:11.104687 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:20:11.104709 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:11.104709 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:11.104718 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:11.104775 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:20:11.104904 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.105415 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.106443 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.106832 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.106892 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:11.106903 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:11.107035 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 13:20:11.107073 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:20:11.107178 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:11.104510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46107 (local address 127.0.98.3:50421)
0925 13:20:11.104599 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:11.104610 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:11.104650 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:11.104788 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:11.104789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:11.104790 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:20:11.104850 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:11.104851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.105484 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.105485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:11.106364 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:11.106367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.106899 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.106899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:11.106978 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:11.106981 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:20:11.107026 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 13:20:11.107073 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":3}
I20260925 13:20:11.107170 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:11.104556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42975 (local address 127.0.98.2:42729)
0925 13:20:11.104721 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:11.104723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:11.104730 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:20:11.104757 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:11.104758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:11.104767 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:11.104816 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:20:11.104942 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.105454 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.106492 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.106873 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.106930 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:11.106941 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:11.107065 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 13:20:11.107102 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260925 13:20:11.107203 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:11.104510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42729 (local address 127.0.98.3:42975)
0925 13:20:11.104674 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:11.104682 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:11.104719 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:11.104820 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:11.104821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:11.104822 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:20:11.104887 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:11.104888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.105612 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.105615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:11.106407 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:11.106409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.106944 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.106945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:11.107009 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:11.107012 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:20:11.107055 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:20:11.107097 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":28,"thread_start_us":90,"threads_started":1}
I20260925 13:20:11.107496 15897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3198795f27784167bbd9b4995e5188d2" candidate_uuid: "a395365b68ce4658ae97b8d4d8e20e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f8f41766d1e44a38206e3c63e115c93" is_pre_election: true
I20260925 13:20:11.107630 15897 raft_consensus.cc:2454] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a395365b68ce4658ae97b8d4d8e20e98 in term 0.
I20260925 13:20:11.107626 16031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3198795f27784167bbd9b4995e5188d2" candidate_uuid: "a395365b68ce4658ae97b8d4d8e20e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c50c01da9426585e416bc06cb627a" is_pre_election: true
I20260925 13:20:11.107771 16031 raft_consensus.cc:2454] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a395365b68ce4658ae97b8d4d8e20e98 in term 0.
I20260925 13:20:11.107862 16098 leader_election.cc:302] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f8f41766d1e44a38206e3c63e115c93, a395365b68ce4658ae97b8d4d8e20e98; no voters:
I20260925 13:20:11.107970 16231 raft_consensus.cc:2781] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:20:11.108029 16231 raft_consensus.cc:482] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:20:11.108052 16231 raft_consensus.cc:3037] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:20:11.108702 16231 raft_consensus.cc:504] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } }
I20260925 13:20:11.108817 16231 leader_election.cc:288] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [CANDIDATE]: Term 1 election: Requested vote from peers 171c50c01da9426585e416bc06cb627a (127.0.98.2:42729), 6f8f41766d1e44a38206e3c63e115c93 (127.0.98.1:46107)
I20260925 13:20:11.108958 16031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3198795f27784167bbd9b4995e5188d2" candidate_uuid: "a395365b68ce4658ae97b8d4d8e20e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c50c01da9426585e416bc06cb627a"
I20260925 13:20:11.108994 15897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3198795f27784167bbd9b4995e5188d2" candidate_uuid: "a395365b68ce4658ae97b8d4d8e20e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f8f41766d1e44a38206e3c63e115c93"
I20260925 13:20:11.109037 16031 raft_consensus.cc:3037] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:20:11.109063 15897 raft_consensus.cc:3037] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:20:11.109799 15897 raft_consensus.cc:2454] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a395365b68ce4658ae97b8d4d8e20e98 in term 1.
I20260925 13:20:11.109798 16031 raft_consensus.cc:2454] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a395365b68ce4658ae97b8d4d8e20e98 in term 1.
I20260925 13:20:11.109963 16101 leader_election.cc:302] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c50c01da9426585e416bc06cb627a, a395365b68ce4658ae97b8d4d8e20e98; no voters:
I20260925 13:20:11.110039 16231 raft_consensus.cc:2781] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:20:11.110177 16231 raft_consensus.cc:686] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [term 1 LEADER]: Becoming Leader. State: Replica: a395365b68ce4658ae97b8d4d8e20e98, State: Running, Role: LEADER
I20260925 13:20:11.110262 16231 consensus_queue.cc:237] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } }
I20260925 13:20:11.111073 15751 catalog_manager.cc:5495] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to a395365b68ce4658ae97b8d4d8e20e98 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a395365b68ce4658ae97b8d4d8e20e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } health_report { overall_health: HEALTHY } } }
I20260925 13:20:11.121294 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:11.119215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.0.98.3:42191)
0925 13:20:11.119263 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:11.119264 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:11.119272 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:20:11.119305 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:11.119306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:11.119324 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:11.119381 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:20:11.119489 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.119960 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.120658 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:11.120857 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:11.120910 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:11.121014 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:20:11.121122 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:20:11.121186 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:11.121198 (+ 12us) server_negotiation.cc:299] Negotiation successful
0925 13:20:11.121215 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:20:11.123091 16031 raft_consensus.cc:1261] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a [term 1 FOLLOWER]: Refusing update from remote peer a395365b68ce4658ae97b8d4d8e20e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:20:11.123091 15897 raft_consensus.cc:1261] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93 [term 1 FOLLOWER]: Refusing update from remote peer a395365b68ce4658ae97b8d4d8e20e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:20:11.123410 16231 consensus_queue.cc:1035] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:20:11.123586 16235 consensus_queue.cc:1035] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:20:11.126128 16239 mvcc.cc:205] Tried to move back new op lower bound from 7333242515957522432 to 7333242515908259840. Current Snapshot: MvccSnapshot[applied={T|T < 7333242515957522432}]
I20260925 13:20:11.128093 15751 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:20:11.128196 15751 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:20:11.128867 15751 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Sending DeleteTablet for 3 replicas of tablet 3198795f27784167bbd9b4995e5188d2
I20260925 13:20:11.129173 16145 tablet_service.cc:1514] Processing DeleteTablet for tablet 3198795f27784167bbd9b4995e5188d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:20:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40798
I20260925 13:20:11.129172 15877 tablet_service.cc:1514] Processing DeleteTablet for tablet 3198795f27784167bbd9b4995e5188d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:20:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58858
I20260925 13:20:11.129412 16011 tablet_service.cc:1514] Processing DeleteTablet for tablet 3198795f27784167bbd9b4995e5188d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:20:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46364
I20260925 13:20:11.129628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 15814
I20260925 13:20:11.129647 16250 tablet_replica.cc:330] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a: stopping tablet replica
I20260925 13:20:11.130901 16250 raft_consensus.cc:2229] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:20:11.131028 16250 raft_consensus.cc:2258] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:20:11.131515 16250 ts_tablet_manager.cc:1905] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:20:11.131662 16249 tablet_replica.cc:330] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98: stopping tablet replica
I20260925 13:20:11.131754 16249 raft_consensus.cc:2229] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:20:11.131858 16249 raft_consensus.cc:2258] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:20:11.132253 16249 ts_tablet_manager.cc:1905] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:20:11.133302 16250 ts_tablet_manager.cc:1918] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:20:11.133375 16250 log.cc:1196] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/3198795f27784167bbd9b4995e5188d2
I20260925 13:20:11.133738 16250 ts_tablet_manager.cc:1939] T 3198795f27784167bbd9b4995e5188d2 P 171c50c01da9426585e416bc06cb627a: Deleting consensus metadata
I20260925 13:20:11.134104 16249 ts_tablet_manager.cc:1918] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:20:11.134174 15739 catalog_manager.cc:4841] TS 171c50c01da9426585e416bc06cb627a (127.0.98.2:42729): tablet 3198795f27784167bbd9b4995e5188d2 (table test-table [id=b9b7ef762cc6430ebe2560b2b2d5a79e]) successfully deleted
I20260925 13:20:11.134302 16249 log.cc:1196] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/3198795f27784167bbd9b4995e5188d2
I20260925 13:20:11.134545 16249 ts_tablet_manager.cc:1939] T 3198795f27784167bbd9b4995e5188d2 P a395365b68ce4658ae97b8d4d8e20e98: Deleting consensus metadata
I20260925 13:20:11.134914 15739 catalog_manager.cc:4841] TS a395365b68ce4658ae97b8d4d8e20e98 (127.0.98.3:42191): tablet 3198795f27784167bbd9b4995e5188d2 (table test-table [id=b9b7ef762cc6430ebe2560b2b2d5a79e]) successfully deleted
W20260925 13:20:11.135620 15736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46107 (error 108)
I20260925 13:20:11.136106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 15948
I20260925 13:20:11.136114 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:11.135973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46107 (local address 127.0.0.1:58870)
0925 13:20:11.136041 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:11.136078 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 13:20:11.136242 15736 catalog_manager.cc:4584] TS 6f8f41766d1e44a38206e3c63e115c93 (127.0.98.1:46107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3198795f27784167bbd9b4995e5188d2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46107: connect: Connection refused (error 111)
I20260925 13:20:11.140606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 16082
I20260925 13:20:11.144673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 15720
I20260925 13:20:31.149492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34629
--webserver_interface=127.0.98.62
--webserver_port=41691
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:20:31.228636 16252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:20:31.228850 16252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:20:31.228871 16252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:20:31.230046 16252 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 13:20:31.230082 16252 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 13:20:31.230093 16252 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 13:20:31.230103 16252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:20:31.230115 16252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:20:31.230127 16252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:20:31.230142 16252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:20:31.230165 16252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:20:31.231614 16252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:20:31.231846 16252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:20:31.232054 16252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:20:31.240773 16252 server_base.cc:1034] running on GCE node
W20260925 13:20:31.240756 16258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:31.240744 16259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:31.240861 16261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:31.241441 16252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:20:31.241813 16252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:20:31.242990 16252 hybrid_clock.cc:648] HybridClock initialized: now 1790342431242979 us; error 26 us; skew 500 ppm
Sep 25 13:20:31 dist-test-slave-t8d9 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:20:31.245419 16252 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:20:31.245855 16252 webserver.cc:460] Webserver started at http://127.0.98.62:41691/ using document root <none> and password file <none>
I20260925 13:20:31.246043 16252 fs_manager.cc:359] Metadata directory not provided
I20260925 13:20:31.246094 16252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:20:31.247370 16252 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:20:31.247972 16268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:31.248135 16252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:20:31.248214 16252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98"
format_stamp: "Formatted at 2026-09-25 13:20:10 on dist-test-slave-t8d9"
server_key: "3bb6fae1afde7ac07739a147bc59969b"
server_key_iv: "2aa63f660f2166242e0bbff4a9e85456"
server_key_version: "encryptionkey@0"
I20260925 13:20:31.248596 16252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:20:31.279465 16252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:20:31.281356 16252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:20:31.281463 16252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:20:31.285092 16252 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34629
I20260925 13:20:31.285135 16320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34629 every 8 connection(s)
I20260925 13:20:31.285485 16252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:20:31.286978 16321 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:20:31.287628 16321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Bootstrap starting.
I20260925 13:20:31.294622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 16252
I20260925 13:20:31.294888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46107
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34629
--webserver_port=37167
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:20:31.295413 16321 log.cc:824] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Log is configured to *not* fsync() on all Append() calls
I20260925 13:20:31.297646 16321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:20:31.297909 16321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Bootstrap complete.
I20260925 13:20:31.300031 16321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260925 13:20:31.300247 16321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3dd3a2fd304aa282576ffa8bd2fc98, State: Initialized, Role: FOLLOWER
I20260925 13:20:31.300392 16321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260925 13:20:31.300462 16321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:20:31.300514 16321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:20:31.300560 16321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 1 FOLLOWER]: Advancing to term 2
I20260925 13:20:31.301522 16321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260925 13:20:31.301643 16321 leader_election.cc:302] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e3dd3a2fd304aa282576ffa8bd2fc98; no voters:
I20260925 13:20:31.301820 16321 leader_election.cc:288] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 13:20:31.301863 16325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 13:20:31.301970 16325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 2 LEADER]: Becoming Leader. State: Replica: 9e3dd3a2fd304aa282576ffa8bd2fc98, State: Running, Role: LEADER
I20260925 13:20:31.302071 16321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:20:31.302052 16325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260925 13:20:31.302372 16326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } } }
I20260925 13:20:31.302456 16326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [sys.catalog]: This master's current role is: LEADER
I20260925 13:20:31.302706 16327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e3dd3a2fd304aa282576ffa8bd2fc98. Latest consensus state: current_term: 2 leader_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } } }
I20260925 13:20:31.302759 16331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:20:31.303372 16327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [sys.catalog]: This master's current role is: LEADER
I20260925 13:20:31.303727 16331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:20:31.304090 16331 catalog_manager.cc:1259] Loaded cluster ID: 0717d2a1aebe495a82fc228ff874c19c
I20260925 13:20:31.304129 16331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:20:31.305336 16331 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:20:31.305562 16331 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Loaded TSK: 0
I20260925 13:20:31.316983 16331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Generated new TSK 1
I20260925 13:20:31.317085 16331 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 13:20:31.378755 16324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:20:31.378942 16324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:20:31.378962 16324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:20:31.380087 16324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:20:31.380125 16324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:20:31.380151 16324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:20:31.381568 16324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37167
--tserver_master_addrs=127.0.98.62:34629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:20:31.381783 16324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:20:31.381995 16324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:20:31.386607 16348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:31.386633 16349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:31.386857 16351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:31.386855 16324 server_base.cc:1034] running on GCE node
I20260925 13:20:31.387063 16324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:20:31.387280 16324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:20:31.388422 16324 hybrid_clock.cc:648] HybridClock initialized: now 1790342431388413 us; error 26 us; skew 500 ppm
Sep 25 13:20:31 dist-test-slave-t8d9 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:20:31.390977 16324 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:20:31.391405 16324 webserver.cc:460] Webserver started at http://127.0.98.1:37167/ using document root <none> and password file <none>
I20260925 13:20:31.391614 16324 fs_manager.cc:359] Metadata directory not provided
I20260925 13:20:31.391676 16324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:20:31.393009 16324 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:20:31.393664 16358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:31.393851 16324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:20:31.393942 16324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "6f8f41766d1e44a38206e3c63e115c93"
format_stamp: "Formatted at 2026-09-25 13:20:10 on dist-test-slave-t8d9"
server_key: "a43ae166631e50146a51f545b68f508a"
server_key_iv: "f1cb3f2cffaf0e666a0a20b271fb7cea"
server_key_version: "encryptionkey@0"
I20260925 13:20:31.394309 16324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:20:31.408116 16324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:20:31.410311 16324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:20:31.410451 16324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:20:31.410656 16324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:20:31.411077 16365 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 13:20:31.412914 16324 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 13:20:31.412977 16324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 13:20:31.413014 16324 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 13:20:31.413753 16324 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 13:20:31.413785 16324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260925 13:20:31.413877 16365 tablet_bootstrap.cc:492] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: Bootstrap starting.
I20260925 13:20:31.419675 16324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46107
I20260925 13:20:31.419909 16472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46107 every 8 connection(s)
I20260925 13:20:31.420226 16324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:20:31.424767 16365 log.cc:824] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: Log is configured to *not* fsync() on all Append() calls
Sep 25 13:20:31 dist-test-slave-t8d9 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:20:31.427613 16365 tablet_bootstrap.cc:492] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:20:31.427928 16365 tablet_bootstrap.cc:492] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: Bootstrap complete.
I20260925 13:20:31.428815 16365 ts_tablet_manager.cc:1397] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260925 13:20:31.429955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 16324
I20260925 13:20:31.429845 16365 raft_consensus.cc:348] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } }
I20260925 13:20:31.430199 16365 raft_consensus.cc:729] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8f41766d1e44a38206e3c63e115c93, State: Initialized, Role: FOLLOWER
I20260925 13:20:31.430253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42729
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34629
--webserver_port=36289
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:20:31.430337 16365 consensus_queue.cc:260] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } }
I20260925 13:20:31.430630 16365 ts_tablet_manager.cc:1428] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
I20260925 13:20:31.432045 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.421035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37215 (local address 127.0.98.62:34629)
0925 13:20:31.421925 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:31.421933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:31.421960 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:20:31.422063 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:31.422068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:31.422108 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:31.422228 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:20:31.422857 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.423699 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.424691 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.425199 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.425311 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.427941 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:20:31.427946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:20:31.427946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:20:31.427958 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:20:31.429690 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:31.430794 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:31.430797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:31.430798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:31.430889 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:31.431131 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:31.431132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:31.431133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:31.431363 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:20:31.431439 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:31.431720 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 13:20:31.431850 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":68,"threads_started":1}
I20260925 13:20:31.432045 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.420958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:37215)
0925 13:20:31.421231 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:31.421268 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:31.421768 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:31.422264 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:31.422268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:31.422547 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:20:31.422743 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.422748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.423818 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.423818 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.424609 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.424611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.425312 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.425313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.425473 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.426098 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:20:31.426112 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:20:31.427835 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:20:31.429772 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:31.429779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:31.429781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:31.430068 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:31.430961 (+ 893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:31.430962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:31.430962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:31.431068 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:31.431429 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:20:31.431436 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:20:31.431569 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 13:20:31.431654 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 13:20:31.431850 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":88,"threads_started":1}
I20260925 13:20:31.432637 16473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34629
I20260925 13:20:31.432732 16473 heartbeater.cc:464] Registering TS with master...
I20260925 13:20:31.432890 16473 heartbeater.cc:511] Master 127.0.98.62:34629 requested a full tablet report, sending...
I20260925 13:20:31.433574 16285 ts_manager.cc:194] Registered new tserver with Master: 6f8f41766d1e44a38206e3c63e115c93 (127.0.98.1:46107)
I20260925 13:20:31.434504 16285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37215
I20260925 13:20:31.435591 16473 heartbeater.cc:503] Master 127.0.98.62:34629 was elected leader, sending a full tablet report...
Sep 25 13:20:31 dist-test-slave-t8d9 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:20:31.441936 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.434159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46107 (local address 127.0.0.1:54244)
0925 13:20:31.434386 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:31.434419 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:31.434482 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:31.434765 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:31.434770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:31.435086 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:20:31.435133 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.435136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.436395 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.436396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.436981 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.436983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.437408 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.437412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.437501 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.437982 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:20:31.437987 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:20:31.439233 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:20:31.440821 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:31.440828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:31.440830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:31.441021 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:31.441210 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:31.441211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:31.441211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:31.441270 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:31.441525 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:20:31.441562 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:20:31.441619 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 13:20:31.441695 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 13:20:31.441828 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260925 13:20:31.441936 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.434225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.0.98.1:46107)
0925 13:20:31.434621 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:31.434625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:31.434634 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:20:31.434667 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:31.434669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:31.434682 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:31.434751 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:20:31.435216 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.436307 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.437059 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.437346 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.437404 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.439321 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:20:31.439329 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:20:31.439330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:20:31.439337 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:20:31.440739 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:31.441094 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:31.441097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:31.441098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:31.441149 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:31.441319 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:31.441320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:31.441320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:31.441464 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:20:31.441570 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:31.441741 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 13:20:31.441833 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":364,"thread_start_us":71,"threads_started":1}
I20260925 13:20:31.442588 16407 tablet_service.cc:1514] Processing DeleteTablet for tablet 3198795f27784167bbd9b4995e5188d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:20:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54244
I20260925 13:20:31.443125 16482 tablet_replica.cc:330] stopping tablet replica
I20260925 13:20:31.443248 16482 raft_consensus.cc:2229] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:20:31.443327 16482 raft_consensus.cc:2258] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:20:31.443655 16482 ts_tablet_manager.cc:1905] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:20:31.446200 16482 ts_tablet_manager.cc:1918] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:20:31.446295 16482 log.cc:1196] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal/wals/3198795f27784167bbd9b4995e5188d2
I20260925 13:20:31.446516 16482 ts_tablet_manager.cc:1939] T 3198795f27784167bbd9b4995e5188d2 P 6f8f41766d1e44a38206e3c63e115c93: Deleting consensus metadata
I20260925 13:20:31.447114 16269 catalog_manager.cc:4841] TS 6f8f41766d1e44a38206e3c63e115c93 (127.0.98.1:46107): tablet 3198795f27784167bbd9b4995e5188d2 (table test-table [id=b9b7ef762cc6430ebe2560b2b2d5a79e]) successfully deleted
W20260925 13:20:31.512614 16479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:20:31.512848 16479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:20:31.512872 16479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:20:31.514014 16479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:20:31.514048 16479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:20:31.514073 16479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:20:31.515455 16479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36289
--tserver_master_addrs=127.0.98.62:34629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:20:31.515650 16479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:20:31.515839 16479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:20:31.520648 16488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:31.520650 16490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:31.520705 16487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:31.520897 16479 server_base.cc:1034] running on GCE node
I20260925 13:20:31.521090 16479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:20:31.521332 16479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:20:31.522480 16479 hybrid_clock.cc:648] HybridClock initialized: now 1790342431522462 us; error 28 us; skew 500 ppm
Sep 25 13:20:31 dist-test-slave-t8d9 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:20:31.524865 16479 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:20:31.525256 16479 webserver.cc:460] Webserver started at http://127.0.98.2:36289/ using document root <none> and password file <none>
I20260925 13:20:31.525439 16479 fs_manager.cc:359] Metadata directory not provided
I20260925 13:20:31.525491 16479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:20:31.526751 16479 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:20:31.527251 16497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:31.527402 16479 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:31.527477 16479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "171c50c01da9426585e416bc06cb627a"
format_stamp: "Formatted at 2026-09-25 13:20:10 on dist-test-slave-t8d9"
server_key: "4ef07e36c66ad90fc3a8fc0bf81d8a41"
server_key_iv: "7d41831090be7bb11bfc987653fc4cd2"
server_key_version: "encryptionkey@0"
I20260925 13:20:31.527799 16479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:20:31.543771 16479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:20:31.545828 16479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:20:31.545941 16479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:20:31.546136 16479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:20:31.546378 16479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:20:31.546406 16479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:31.546427 16479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:20:31.546437 16479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:31.551697 16479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42729
I20260925 13:20:31.551764 16610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42729 every 8 connection(s)
I20260925 13:20:31.552084 16479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:20:31.555277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 16479
I20260925 13:20:31.555562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42191
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34629
--webserver_port=35183
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 13:20:31 dist-test-slave-t8d9 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:20:31.562284 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.552846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46495 (local address 127.0.98.62:34629)
0925 13:20:31.552914 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:31.552917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:31.553133 (+ 216us) server_negotiation.cc:407] Connection header received
0925 13:20:31.553667 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:31.553669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:31.553678 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:31.553742 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:20:31.554702 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.555311 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.556409 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.556701 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.556777 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.559248 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:20:31.559250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:20:31.559251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:20:31.559260 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:20:31.560910 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:31.561310 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:31.561311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:31.561312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:31.561375 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:31.561631 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:31.561632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:31.561632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:31.561785 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:20:31.561856 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:31.562097 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 13:20:31.562186 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:20:31.562369 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.552853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:46495)
0925 13:20:31.553016 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:31.553053 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:31.553522 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:31.553814 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:31.553817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:31.554389 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:20:31.554590 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.554596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.555426 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.555428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.556322 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.556324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.556816 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.556818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.556976 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.557517 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:20:31.557556 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:20:31.559143 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:20:31.561000 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:31.561007 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:31.561008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:31.561241 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:31.561437 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:31.561437 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:31.561438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:31.561569 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:31.561843 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:20:31.561846 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:20:31.561972 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 13:20:31.562048 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 13:20:31.562175 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260925 13:20:31.562840 16611 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34629
I20260925 13:20:31.562932 16611 heartbeater.cc:464] Registering TS with master...
I20260925 13:20:31.563108 16611 heartbeater.cc:511] Master 127.0.98.62:34629 requested a full tablet report, sending...
I20260925 13:20:31.563499 16285 ts_manager.cc:194] Registered new tserver with Master: 171c50c01da9426585e416bc06cb627a (127.0.98.2:42729)
I20260925 13:20:31.564005 16285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46495
W20260925 13:20:31.637451 16614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:20:31.637698 16614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:20:31.637727 16614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:20:31.638872 16614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:20:31.638909 16614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:20:31.638933 16614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:20:31.640352 16614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35183
--tserver_master_addrs=127.0.98.62:34629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:20:31.640537 16614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:20:31.640718 16614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:20:31.645416 16620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:31.645438 16622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:31.645478 16614 server_base.cc:1034] running on GCE node
W20260925 13:20:31.645453 16619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:31.645817 16614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:20:31.646011 16614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:20:31.647142 16614 hybrid_clock.cc:648] HybridClock initialized: now 1790342431647130 us; error 28 us; skew 500 ppm
Sep 25 13:20:31 dist-test-slave-t8d9 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:20:31.649425 16614 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:20:31.649848 16614 webserver.cc:460] Webserver started at http://127.0.98.3:35183/ using document root <none> and password file <none>
I20260925 13:20:31.650038 16614 fs_manager.cc:359] Metadata directory not provided
I20260925 13:20:31.650098 16614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:20:31.651412 16614 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:20:31.651963 16629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:31.652104 16614 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:20:31.652179 16614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "a395365b68ce4658ae97b8d4d8e20e98"
format_stamp: "Formatted at 2026-09-25 13:20:11 on dist-test-slave-t8d9"
server_key: "0abed15cd72af8e6f74a5a4690512628"
server_key_iv: "e8f3c42313d82e27f7f476d53ecb44ce"
server_key_version: "encryptionkey@0"
I20260925 13:20:31.652550 16614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:20:31.672991 16614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:20:31.675109 16614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:20:31.675246 16614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:20:31.675475 16614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:20:31.675765 16614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:20:31.675798 16614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:31.675827 16614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:20:31.675850 16614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:31.681137 16614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42191
I20260925 13:20:31.681211 16742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42191 every 8 connection(s)
I20260925 13:20:31.681510 16614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:20:31 dist-test-slave-t8d9 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:20:31.690552 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.682209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48121 (local address 127.0.98.62:34629)
0925 13:20:31.682285 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:31.682294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:31.682480 (+ 186us) server_negotiation.cc:407] Connection header received
0925 13:20:31.682956 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:31.682958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:31.682968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:31.683017 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:20:31.683922 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.684414 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.685405 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.685655 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.685753 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.687861 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:20:31.687864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:20:31.687864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:20:31.687874 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:20:31.689171 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:31.689618 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:31.689620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:31.689620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:31.689678 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:31.689923 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:31.689925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:31.689925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:31.690038 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:20:31.690094 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:31.690372 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 13:20:31.690457 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 13:20:31.690734 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.682204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:48121)
0925 13:20:31.682368 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:31.682404 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:31.682880 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:31.683084 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:31.683087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:31.683638 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:20:31.683841 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.683846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.684482 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.684483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.685288 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.685290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.685756 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.685756 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.685874 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.686324 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:20:31.686332 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:20:31.687758 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:20:31.689270 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:31.689279 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:31.689282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:31.689513 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:31.689744 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:31.689745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:31.689745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:31.689863 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:31.690093 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:20:31.690101 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:20:31.690251 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 13:20:31.690344 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 13:20:31.690514 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260925 13:20:31.691118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 16614
I20260925 13:20:31.691203 16743 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34629
I20260925 13:20:31.691306 16743 heartbeater.cc:464] Registering TS with master...
I20260925 13:20:31.691500 16743 heartbeater.cc:511] Master 127.0.98.62:34629 requested a full tablet report, sending...
I20260925 13:20:31.691953 16285 ts_manager.cc:194] Registered new tserver with Master: a395365b68ce4658ae97b8d4d8e20e98 (127.0.98.3:42191)
I20260925 13:20:31.692317 16285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48121
I20260925 13:20:31.697147 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.691482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34438 (local address 127.0.98.62:34629)
0925 13:20:31.691521 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:31.691523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:31.691720 (+ 197us) server_negotiation.cc:407] Connection header received
0925 13:20:31.691760 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:31.691761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:31.691771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:31.691842 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:20:31.692398 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.692925 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.693706 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.693902 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.693954 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.695026 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:20:31.695027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:20:31.695027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:20:31.695034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:20:31.696195 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:31.696485 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:31.696486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:31.696487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:31.696535 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:31.696693 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:31.696693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:31.696694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:31.696797 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:20:31.696955 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:31.696990 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 13:20:31.697061 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:20:31.697845 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:20:31.700402 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.698139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.0.98.62:34629)
0925 13:20:31.698204 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:31.698206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:31.698288 (+ 82us) server_negotiation.cc:407] Connection header received
0925 13:20:31.698358 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:31.698359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:31.698400 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:31.698458 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:20:31.698560 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.699019 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.699746 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.699942 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.699993 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.700127 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:20:31.700234 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:20:31.700304 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:31.700323 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 13:20:31.700340 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:20:31.700639 16285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:34450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:20:31.702309 16285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:20:31.704326 16407 tablet_service.cc:1467] Processing CreateTablet for tablet c6225a690eee4f54bff200c4683f813f (DEFAULT_TABLE table=test-table [id=2320c0e442e44c3abdb62b6cdf2eba67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:20:31.704506 16407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6225a690eee4f54bff200c4683f813f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:20:31.706017 16365 tablet_bootstrap.cc:492] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93: Bootstrap starting.
I20260925 13:20:31.706305 16365 tablet_bootstrap.cc:654] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93: Neither blocks nor log segments found. Creating new log.
I20260925 13:20:31.707039 16365 tablet_bootstrap.cc:492] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93: No bootstrap required, opened a new log
I20260925 13:20:31.707086 16365 ts_tablet_manager.cc:1397] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 13:20:31.707229 16365 raft_consensus.cc:348] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:31.707300 16365 raft_consensus.cc:374] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:20:31.707345 16365 raft_consensus.cc:729] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8f41766d1e44a38206e3c63e115c93, State: Initialized, Role: FOLLOWER
I20260925 13:20:31.707383 16365 consensus_queue.cc:260] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:31.707466 16365 ts_tablet_manager.cc:1428] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:31.707993 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.704005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42729 (local address 127.0.0.1:49186)
0925 13:20:31.704183 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:31.704192 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:31.704230 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:31.704505 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:31.704506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:31.704508 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:20:31.704552 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.704553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.705755 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.705756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.706738 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.706741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.707721 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.707722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.707802 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.707855 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 13:20:31.707897 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 13:20:31.707942 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":43}
I20260925 13:20:31.708031 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.704107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.0.98.2:42729)
0925 13:20:31.704342 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:31.704347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:31.704355 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:20:31.704386 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:31.704388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:31.704399 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:31.704489 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:20:31.704641 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.705687 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.706821 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.707636 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.707820 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.707886 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:31.707914 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 13:20:31.707950 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":114,"threads_started":1}
I20260925 13:20:31.708786 16545 tablet_service.cc:1467] Processing CreateTablet for tablet c6225a690eee4f54bff200c4683f813f (DEFAULT_TABLE table=test-table [id=2320c0e442e44c3abdb62b6cdf2eba67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:20:31.709076 16545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6225a690eee4f54bff200c4683f813f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:20:31.709486 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.704094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42191 (local address 127.0.0.1:46362)
0925 13:20:31.704187 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:31.704195 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:31.704232 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:31.706387 (+ 2155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:31.706388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:31.706391 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:20:31.706437 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.706438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.707704 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.707705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.708569 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.708571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.709243 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.709244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.709325 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.709329 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:20:31.709379 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 13:20:31.709415 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":55,"threads_started":1}
I20260925 13:20:31.709582 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.706052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.0.98.3:42191)
0925 13:20:31.706220 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:31.706224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:31.706233 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:20:31.706271 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:31.706273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:31.706288 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:31.706365 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:20:31.706508 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.707618 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.708654 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.709170 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.709229 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.709282 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:31.709437 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 13:20:31.709482 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260925 13:20:31.710244 16677 tablet_service.cc:1467] Processing CreateTablet for tablet c6225a690eee4f54bff200c4683f813f (DEFAULT_TABLE table=test-table [id=2320c0e442e44c3abdb62b6cdf2eba67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:20:31.710579 16677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6225a690eee4f54bff200c4683f813f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:20:31.712430 16752 tablet_bootstrap.cc:492] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a: Bootstrap starting.
I20260925 13:20:31.713035 16752 tablet_bootstrap.cc:654] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a: Neither blocks nor log segments found. Creating new log.
I20260925 13:20:31.713477 16752 log.cc:824] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a: Log is configured to *not* fsync() on all Append() calls
I20260925 13:20:31.713647 16753 tablet_bootstrap.cc:492] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: Bootstrap starting.
I20260925 13:20:31.714129 16752 tablet_bootstrap.cc:492] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a: No bootstrap required, opened a new log
I20260925 13:20:31.714145 16753 tablet_bootstrap.cc:654] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: Neither blocks nor log segments found. Creating new log.
I20260925 13:20:31.714183 16752 ts_tablet_manager.cc:1397] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:20:31.714576 16753 log.cc:824] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: Log is configured to *not* fsync() on all Append() calls
I20260925 13:20:31.715174 16753 tablet_bootstrap.cc:492] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: No bootstrap required, opened a new log
I20260925 13:20:31.715232 16753 ts_tablet_manager.cc:1397] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:20:31.715576 16752 raft_consensus.cc:348] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:31.715760 16752 raft_consensus.cc:374] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:20:31.715790 16752 raft_consensus.cc:729] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c50c01da9426585e416bc06cb627a, State: Initialized, Role: FOLLOWER
I20260925 13:20:31.715904 16752 consensus_queue.cc:260] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:31.716148 16752 ts_tablet_manager.cc:1428] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:20:31.716168 16611 heartbeater.cc:503] Master 127.0.98.62:34629 was elected leader, sending a full tablet report...
I20260925 13:20:31.716975 16753 raft_consensus.cc:348] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:31.717134 16753 raft_consensus.cc:374] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:20:31.717170 16753 raft_consensus.cc:729] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a395365b68ce4658ae97b8d4d8e20e98, State: Initialized, Role: FOLLOWER
I20260925 13:20:31.717274 16753 consensus_queue.cc:260] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:31.717495 16753 ts_tablet_manager.cc:1428] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:20:31.717552 16743 heartbeater.cc:503] Master 127.0.98.62:34629 was elected leader, sending a full tablet report...
I20260925 13:20:31.763885 16478 raft_consensus.cc:482] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:20:31.764060 16478 raft_consensus.cc:504] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:31.764365 16478 leader_election.cc:288] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c50c01da9426585e416bc06cb627a (127.0.98.2:42729), a395365b68ce4658ae97b8d4d8e20e98 (127.0.98.3:42191)
I20260925 13:20:31.767127 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.764528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42729 (local address 127.0.98.1:42237)
0925 13:20:31.764578 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:31.764590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:31.764648 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:31.764808 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:31.764810 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:31.764811 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:20:31.764879 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.764881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.765493 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.765494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.766344 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.766346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.766869 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.766870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.766950 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.766969 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 13:20:31.767014 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 13:20:31.767051 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 13:20:31.767127 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.764622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42237 (local address 127.0.98.2:42729)
0925 13:20:31.764668 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:31.764669 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:31.764682 (+ 13us) server_negotiation.cc:407] Connection header received
0925 13:20:31.764711 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:31.764712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:31.764724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:31.764791 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:20:31.764942 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.765425 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.766431 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.766799 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.766859 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.766870 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:31.767030 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 13:20:31.767066 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:20:31.767236 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.764742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55675 (local address 127.0.98.3:42191)
0925 13:20:31.764824 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:31.764827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:31.764900 (+ 73us) server_negotiation.cc:407] Connection header received
0925 13:20:31.764941 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:31.764942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:31.764953 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:31.764999 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:20:31.765137 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.765646 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.766564 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.766934 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.766990 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.766999 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:31.767142 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 13:20:31.767176 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 13:20:31.767257 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.764648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42191 (local address 127.0.98.1:55675)
0925 13:20:31.764841 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:31.764850 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:31.764897 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:31.765006 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:31.765006 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:31.765008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:20:31.765083 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.765084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.765706 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.765707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.766483 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:31.766485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.767018 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.767019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:31.767084 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.767087 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:20:31.767130 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:20:31.767167 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":111,"threads_started":1}
I20260925 13:20:31.767468 16565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6225a690eee4f54bff200c4683f813f" candidate_uuid: "6f8f41766d1e44a38206e3c63e115c93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c50c01da9426585e416bc06cb627a" is_pre_election: true
I20260925 13:20:31.767607 16565 raft_consensus.cc:2454] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f8f41766d1e44a38206e3c63e115c93 in term 0.
I20260925 13:20:31.767715 16697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6225a690eee4f54bff200c4683f813f" candidate_uuid: "6f8f41766d1e44a38206e3c63e115c93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a395365b68ce4658ae97b8d4d8e20e98" is_pre_election: true
I20260925 13:20:31.767856 16697 raft_consensus.cc:2454] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f8f41766d1e44a38206e3c63e115c93 in term 0.
I20260925 13:20:31.767838 16362 leader_election.cc:302] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c50c01da9426585e416bc06cb627a, 6f8f41766d1e44a38206e3c63e115c93; no voters:
I20260925 13:20:31.767961 16478 raft_consensus.cc:2781] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:20:31.768033 16478 raft_consensus.cc:482] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:20:31.768057 16478 raft_consensus.cc:3037] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:20:31.768720 16478 raft_consensus.cc:504] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:31.768824 16478 leader_election.cc:288] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [CANDIDATE]: Term 1 election: Requested vote from peers 171c50c01da9426585e416bc06cb627a (127.0.98.2:42729), a395365b68ce4658ae97b8d4d8e20e98 (127.0.98.3:42191)
I20260925 13:20:31.768994 16565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6225a690eee4f54bff200c4683f813f" candidate_uuid: "6f8f41766d1e44a38206e3c63e115c93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c50c01da9426585e416bc06cb627a"
I20260925 13:20:31.769032 16697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6225a690eee4f54bff200c4683f813f" candidate_uuid: "6f8f41766d1e44a38206e3c63e115c93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a395365b68ce4658ae97b8d4d8e20e98"
I20260925 13:20:31.769075 16565 raft_consensus.cc:3037] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:20:31.769090 16697 raft_consensus.cc:3037] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:20:31.769807 16697 raft_consensus.cc:2454] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f8f41766d1e44a38206e3c63e115c93 in term 1.
I20260925 13:20:31.769827 16565 raft_consensus.cc:2454] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f8f41766d1e44a38206e3c63e115c93 in term 1.
I20260925 13:20:31.769954 16362 leader_election.cc:302] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c50c01da9426585e416bc06cb627a, 6f8f41766d1e44a38206e3c63e115c93; no voters:
I20260925 13:20:31.770056 16478 raft_consensus.cc:2781] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:20:31.770219 16478 raft_consensus.cc:686] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 1 LEADER]: Becoming Leader. State: Replica: 6f8f41766d1e44a38206e3c63e115c93, State: Running, Role: LEADER
I20260925 13:20:31.770283 16478 consensus_queue.cc:237] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:31.771031 16285 catalog_manager.cc:5495] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f8f41766d1e44a38206e3c63e115c93 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6f8f41766d1e44a38206e3c63e115c93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } health_report { overall_health: HEALTHY } } }
I20260925 13:20:31.780699 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.778697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.0.98.1:46107)
0925 13:20:31.778751 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:31.778752 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:31.778759 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:20:31.778790 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:31.778791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:31.778805 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:31.778860 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:20:31.778962 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.779434 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.780085 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:31.780285 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:31.780331 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:31.780431 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:20:31.780524 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:20:31.780571 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:31.780601 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 13:20:31.780618 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:20:31.782130 16697 raft_consensus.cc:1261] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98 [term 1 FOLLOWER]: Refusing update from remote peer 6f8f41766d1e44a38206e3c63e115c93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:20:31.782136 16565 raft_consensus.cc:1261] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 1 FOLLOWER]: Refusing update from remote peer 6f8f41766d1e44a38206e3c63e115c93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:20:31.782444 16478 consensus_queue.cc:1035] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:20:31.782573 16759 consensus_queue.cc:1035] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:20:31.787030 16284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:34450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:20:31.787129 16284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:34450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:20:31.787835 16284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Sending DeleteTablet for 3 replicas of tablet c6225a690eee4f54bff200c4683f813f
I20260925 13:20:31.788098 16407 tablet_service.cc:1514] Processing DeleteTablet for tablet c6225a690eee4f54bff200c4683f813f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:20:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54244
I20260925 13:20:31.788228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 16324
I20260925 13:20:31.788216 16545 tablet_service.cc:1514] Processing DeleteTablet for tablet c6225a690eee4f54bff200c4683f813f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:20:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49186
I20260925 13:20:31.788302 16677 tablet_service.cc:1514] Processing DeleteTablet for tablet c6225a690eee4f54bff200c4683f813f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:20:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46362
I20260925 13:20:31.788431 16482 tablet_replica.cc:330] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93: stopping tablet replica
I20260925 13:20:31.788481 16482 raft_consensus.cc:2229] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:20:31.788549 16482 raft_consensus.cc:2258] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:20:31.788791 16773 tablet_replica.cc:330] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: stopping tablet replica
I20260925 13:20:31.788869 16773 raft_consensus.cc:2229] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:20:31.788942 16773 raft_consensus.cc:2258] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:20:31.789299 16773 ts_tablet_manager.cc:1905] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:20:31.790871 16773 ts_tablet_manager.cc:1918] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:20:31.790938 16773 log.cc:1196] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/c6225a690eee4f54bff200c4683f813f
I20260925 13:20:31.791153 16773 ts_tablet_manager.cc:1939] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: Deleting consensus metadata
I20260925 13:20:31.791503 16272 catalog_manager.cc:4841] TS a395365b68ce4658ae97b8d4d8e20e98 (127.0.98.3:42191): tablet c6225a690eee4f54bff200c4683f813f (table test-table [id=2320c0e442e44c3abdb62b6cdf2eba67]) successfully deleted
W20260925 13:20:31.793160 16269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46107 (error 108)
I20260925 13:20:31.793624 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.793491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46107 (local address 127.0.0.1:54256)
0925 13:20:31.793559 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:31.793593 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 13:20:31.793692 16772 tablet_replica.cc:330] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a: stopping tablet replica
I20260925 13:20:31.793803 16772 raft_consensus.cc:2229] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:20:31.793915 16772 raft_consensus.cc:2258] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 13:20:31.793725 16269 catalog_manager.cc:4584] TS 6f8f41766d1e44a38206e3c63e115c93 (127.0.98.1:46107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6225a690eee4f54bff200c4683f813f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46107: connect: Connection refused (error 111)
I20260925 13:20:31.794153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 16479
I20260925 13:20:31.798475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 16614
I20260925 13:20:31.798494 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:31.798399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42729 (local address 127.0.0.1:49194)
0925 13:20:31.798442 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:31.798466 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 13:20:31.802591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 16252
I20260925 13:20:51.807621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34629
--webserver_interface=127.0.98.62
--webserver_port=41691
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:20:51.888201 16775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:20:51.888428 16775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:20:51.888460 16775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:20:51.889688 16775 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 13:20:51.889729 16775 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 13:20:51.889753 16775 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 13:20:51.889768 16775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:20:51.889784 16775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:20:51.889803 16775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:20:51.889823 16775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:20:51.889846 16775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:20:51.891340 16775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:20:51.891588 16775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:20:51.891793 16775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:20:51.896714 16775 server_base.cc:1034] running on GCE node
W20260925 13:20:51.896732 16780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:51.896674 16783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:51.896696 16781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:51.897167 16775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:20:51.897413 16775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:20:51.898556 16775 hybrid_clock.cc:648] HybridClock initialized: now 1790342451898553 us; error 42 us; skew 500 ppm
Sep 25 13:20:51 dist-test-slave-t8d9 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:20:51.900844 16775 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:20:51.901289 16775 webserver.cc:460] Webserver started at http://127.0.98.62:41691/ using document root <none> and password file <none>
I20260925 13:20:51.901489 16775 fs_manager.cc:359] Metadata directory not provided
I20260925 13:20:51.901549 16775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:20:51.902988 16775 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:20:51.903651 16790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:51.903829 16775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:20:51.903913 16775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98"
format_stamp: "Formatted at 2026-09-25 13:20:10 on dist-test-slave-t8d9"
server_key: "3bb6fae1afde7ac07739a147bc59969b"
server_key_iv: "2aa63f660f2166242e0bbff4a9e85456"
server_key_version: "encryptionkey@0"
I20260925 13:20:51.904253 16775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:20:51.927935 16775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:20:51.929803 16775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:20:51.929913 16775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:20:51.933506 16775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34629
I20260925 13:20:51.933571 16842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34629 every 8 connection(s)
I20260925 13:20:51.933936 16775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:20:51.935328 16843 sys_catalog.cc:263] Verifying existing consensus state
I20260925 13:20:51.935972 16843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Bootstrap starting.
I20260925 13:20:51.942688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 16775
I20260925 13:20:51.942986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46107
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34629
--webserver_port=37167
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:20:51.944052 16843 log.cc:824] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Log is configured to *not* fsync() on all Append() calls
I20260925 13:20:51.946360 16843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:20:51.946650 16843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Bootstrap complete.
I20260925 13:20:51.948715 16843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260925 13:20:51.948945 16843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3dd3a2fd304aa282576ffa8bd2fc98, State: Initialized, Role: FOLLOWER
I20260925 13:20:51.949095 16843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260925 13:20:51.949170 16843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:20:51.949219 16843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:20:51.949270 16843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 2 FOLLOWER]: Advancing to term 3
I20260925 13:20:51.950335 16843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260925 13:20:51.950441 16843 leader_election.cc:302] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e3dd3a2fd304aa282576ffa8bd2fc98; no voters:
I20260925 13:20:51.950599 16843 leader_election.cc:288] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 13:20:51.950649 16847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 13:20:51.950770 16847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [term 3 LEADER]: Becoming Leader. State: Replica: 9e3dd3a2fd304aa282576ffa8bd2fc98, State: Running, Role: LEADER
I20260925 13:20:51.950870 16843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:20:51.950857 16847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } }
I20260925 13:20:51.951164 16848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } } }
I20260925 13:20:51.951254 16848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [sys.catalog]: This master's current role is: LEADER
I20260925 13:20:51.951459 16852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:20:51.951824 16849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e3dd3a2fd304aa282576ffa8bd2fc98. Latest consensus state: current_term: 3 leader_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd3a2fd304aa282576ffa8bd2fc98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34629 } } }
I20260925 13:20:51.951960 16849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98 [sys.catalog]: This master's current role is: LEADER
I20260925 13:20:51.952292 16852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:20:51.952400 16852 catalog_manager.cc:1259] Loaded cluster ID: 0717d2a1aebe495a82fc228ff874c19c
I20260925 13:20:51.952433 16852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:20:51.953295 16852 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:20:51.953464 16852 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Loaded TSK: 0
I20260925 13:20:51.953500 16852 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9e3dd3a2fd304aa282576ffa8bd2fc98: Loaded TSK: 1
F20260925 13:20:51.953728 16852 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbdef6fc9ad google::LogMessage::Fail() at ??:0
@ 0x7fbdef6ff88a google::LogMessage::SendToLog() at ??:0
@ 0x7fbdef6fc4de google::LogMessage::Flush() at ??:0
@ 0x7fbdef6fda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbdf1e69104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbdf1e66d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fbdf76561cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fbdf765648e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fbdf7656594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fbdf761cc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fbdf7626ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbdf15ca0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbdf15bf954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbdf24236db start_thread at ??:0
@ 0x7fbdee1fa61f clone at ??:0
W20260925 13:20:52.037096 16845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:20:52.037354 16845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:20:52.037389 16845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:20:52.038637 16845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:20:52.038683 16845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:20:52.038718 16845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:20:52.040221 16845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37167
--tserver_master_addrs=127.0.98.62:34629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:20:52.040479 16845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:20:52.040704 16845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:20:52.045596 16884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:52.045615 16845 server_base.cc:1034] running on GCE node
W20260925 13:20:52.045691 16885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:52.045591 16887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:52.045955 16845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:20:52.046164 16845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:20:52.047366 16845 hybrid_clock.cc:648] HybridClock initialized: now 1790342452047342 us; error 37 us; skew 500 ppm
Sep 25 13:20:52 dist-test-slave-t8d9 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:20:52.049892 16845 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:20:52.050313 16845 webserver.cc:460] Webserver started at http://127.0.98.1:37167/ using document root <none> and password file <none>
I20260925 13:20:52.050529 16845 fs_manager.cc:359] Metadata directory not provided
I20260925 13:20:52.050577 16845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:20:52.051929 16845 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:20:52.052604 16894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:52.052758 16845 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:20:52.052876 16845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "6f8f41766d1e44a38206e3c63e115c93"
format_stamp: "Formatted at 2026-09-25 13:20:10 on dist-test-slave-t8d9"
server_key: "a43ae166631e50146a51f545b68f508a"
server_key_iv: "f1cb3f2cffaf0e666a0a20b271fb7cea"
server_key_version: "encryptionkey@0"
I20260925 13:20:52.053231 16845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:20:52.079198 16845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:20:52.081413 16845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:20:52.081586 16845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:20:52.081825 16845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:20:52.082259 16901 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 13:20:52.084215 16845 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 13:20:52.084276 16845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 13:20:52.084311 16845 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 13:20:52.085026 16845 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 13:20:52.085062 16845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 13:20:52.085158 16901 tablet_bootstrap.cc:492] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93: Bootstrap starting.
I20260925 13:20:52.091802 16845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46107
I20260925 13:20:52.091948 17008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46107 every 8 connection(s)
I20260925 13:20:52.092226 16845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:20:52.095976 16901 log.cc:824] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93: Log is configured to *not* fsync() on all Append() calls
I20260925 13:20:52.098168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 16845
I20260925 13:20:52.098441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42729
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34629
--webserver_port=36289
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:20:52.098747 16901 tablet_bootstrap.cc:492] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:20:52.099943 16901 tablet_bootstrap.cc:492] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93: Bootstrap complete.
I20260925 13:20:52.100945 16901 ts_tablet_manager.cc:1397] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260925 13:20:52.102082 16901 raft_consensus.cc:348] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:52.102362 16901 raft_consensus.cc:729] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8f41766d1e44a38206e3c63e115c93, State: Initialized, Role: FOLLOWER
I20260925 13:20:52.102509 16901 consensus_queue.cc:260] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:52.102785 16901 ts_tablet_manager.cc:1428] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260925 13:20:52.162983 17011 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:20:52.092842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:44053)
0925 13:20:52.093008 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:52.093053 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:52.093570 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:52.162770 (+ 69200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
W20260925 13:20:52.163254 17009 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34629 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34629: Client connection negotiation failed: client connection to 127.0.98.62:34629: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 13:20:52.163637 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:52.163490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:46595)
0925 13:20:52.163564 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:52.163610 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 13:20:52.163990 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:52.163901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:46337)
0925 13:20:52.163944 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:52.163968 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 13:20:52.164098 17009 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 13:20:52.194902 17013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:20:52.195111 17013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:20:52.195133 17013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:20:52.196236 17013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:20:52.196270 17013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:20:52.196295 17013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:20:52.197782 17013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36289
--tserver_master_addrs=127.0.98.62:34629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:20:52.197978 17013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:20:52.198160 17013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:20:52.202994 17022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:52.202994 17020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:52.203027 17019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:52.203236 17013 server_base.cc:1034] running on GCE node
I20260925 13:20:52.203404 17013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:20:52.203613 17013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:20:52.204746 17013 hybrid_clock.cc:648] HybridClock initialized: now 1790342452204706 us; error 58 us; skew 500 ppm
Sep 25 13:20:52 dist-test-slave-t8d9 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:20:52.207326 17013 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:20:52.207752 17013 webserver.cc:460] Webserver started at http://127.0.98.2:36289/ using document root <none> and password file <none>
I20260925 13:20:52.207934 17013 fs_manager.cc:359] Metadata directory not provided
I20260925 13:20:52.207976 17013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:20:52.209255 17013 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:20:52.209884 17029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:52.210032 17013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:52.210114 17013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "171c50c01da9426585e416bc06cb627a"
format_stamp: "Formatted at 2026-09-25 13:20:10 on dist-test-slave-t8d9"
server_key: "4ef07e36c66ad90fc3a8fc0bf81d8a41"
server_key_iv: "7d41831090be7bb11bfc987653fc4cd2"
server_key_version: "encryptionkey@0"
I20260925 13:20:52.210460 17013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:20:52.239378 17013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:20:52.241616 17013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:20:52.241771 17013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:20:52.241997 17013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:20:52.242427 17036 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 13:20:52.243932 17013 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 13:20:52.244009 17013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 13:20:52.244052 17013 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 13:20:52.244804 17013 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 13:20:52.244845 17013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 13:20:52.244903 17036 tablet_bootstrap.cc:492] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a: Bootstrap starting.
I20260925 13:20:52.251564 17013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42729
I20260925 13:20:52.251667 17143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42729 every 8 connection(s)
I20260925 13:20:52.252036 17013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:20:52.253235 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:52.252804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:46637)
0925 13:20:52.252987 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:52.253078 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
W20260925 13:20:52.253487 17144 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34629 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34629: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
I20260925 13:20:52.253909 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:52.253715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:32891)
0925 13:20:52.253768 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:52.253884 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 13:20:52.254221 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:52.254156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:40923)
0925 13:20:52.254185 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:52.254206 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260925 13:20:52.254343 17144 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 13:20:52.254591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 17013
I20260925 13:20:52.254855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42191
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34629
--webserver_port=35183
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:20:52.254913 17036 log.cc:824] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a: Log is configured to *not* fsync() on all Append() calls
I20260925 13:20:52.257525 17036 tablet_bootstrap.cc:492] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:20:52.257807 17036 tablet_bootstrap.cc:492] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a: Bootstrap complete.
I20260925 13:20:52.258589 17036 ts_tablet_manager.cc:1397] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260925 13:20:52.259835 17036 raft_consensus.cc:348] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:52.260113 17036 raft_consensus.cc:729] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c50c01da9426585e416bc06cb627a, State: Initialized, Role: FOLLOWER
I20260925 13:20:52.260259 17036 consensus_queue.cc:260] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:52.260550 17036 ts_tablet_manager.cc:1428] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260925 13:20:52.260916 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:52.260758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:45999)
0925 13:20:52.260809 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:52.260852 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 13:20:52.261287 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:52.261177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:49409)
0925 13:20:52.261228 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:52.261261 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 13:20:52.338125 17148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:20:52.338326 17148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:20:52.338349 17148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:20:52.339532 17148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:20:52.339574 17148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:20:52.339599 17148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:20:52.341025 17148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35183
--tserver_master_addrs=127.0.98.62:34629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:20:52.341240 17148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:20:52.341445 17148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:20:52.346164 17157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:52.346174 17154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:20:52.346174 17155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:20:52.346251 17148 server_base.cc:1034] running on GCE node
I20260925 13:20:52.346581 17148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:20:52.346814 17148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:20:52.347950 17148 hybrid_clock.cc:648] HybridClock initialized: now 1790342452347928 us; error 37 us; skew 500 ppm
Sep 25 13:20:52 dist-test-slave-t8d9 krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:20:52.350411 17148 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:20:52.350804 17148 webserver.cc:460] Webserver started at http://127.0.98.3:35183/ using document root <none> and password file <none>
I20260925 13:20:52.350986 17148 fs_manager.cc:359] Metadata directory not provided
I20260925 13:20:52.351025 17148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:20:52.352308 17148 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:20:52.352854 17164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:52.353008 17148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:20:52.353087 17148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "a395365b68ce4658ae97b8d4d8e20e98"
format_stamp: "Formatted at 2026-09-25 13:20:11 on dist-test-slave-t8d9"
server_key: "0abed15cd72af8e6f74a5a4690512628"
server_key_iv: "e8f3c42313d82e27f7f476d53ecb44ce"
server_key_version: "encryptionkey@0"
I20260925 13:20:52.353420 17148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:20:52.370685 17148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:20:52.372653 17148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:20:52.372808 17148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:20:52.373026 17148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:20:52.373319 17148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:20:52.373356 17148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:52.373379 17148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:20:52.373390 17148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:20:52.378798 17148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42191
I20260925 13:20:52.378866 17277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42191 every 8 connection(s)
I20260925 13:20:52.379181 17148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:20:52.380182 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:52.379813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:33291)
0925 13:20:52.379969 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:52.380041 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
W20260925 13:20:52.380542 17278 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34629 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34629: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
I20260925 13:20:52.380779 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:52.380697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:33225)
0925 13:20:52.380737 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:52.380762 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 13:20:52.380975 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:52.380918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:46249)
0925 13:20:52.380946 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:52.380963 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 13:20:52.381050 17278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 13:20:52.383443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 17148
I20260925 13:20:53.164749 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:53.164461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:42929)
0925 13:20:53.164606 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:53.164664 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260925 13:20:53.262050 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:53.261800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:59831)
0925 13:20:53.261942 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:53.261986 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260925 13:20:53.381580 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:53.381356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:44307)
0925 13:20:53.381468 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:53.381503 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260925 13:20:53.486778 17285 raft_consensus.cc:482] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:20:53.486915 17285 raft_consensus.cc:504] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:53.487277 17285 leader_election.cc:288] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a395365b68ce4658ae97b8d4d8e20e98 (127.0.98.3:42191), 6f8f41766d1e44a38206e3c63e115c93 (127.0.98.1:46107)
Sep 25 13:20:53 dist-test-slave-t8d9 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 25 13:20:53 dist-test-slave-t8d9 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 13:20:53.496575 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:53.487607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45811 (local address 127.0.98.3:42191)
0925 13:20:53.487758 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:53.487768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:53.487796 (+ 28us) server_negotiation.cc:407] Connection header received
0925 13:20:53.488244 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:53.488247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:53.488284 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:53.488392 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:20:53.488706 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:53.489573 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:53.490583 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:53.490982 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:53.491075 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:53.493155 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:20:53.493159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:20:53.493159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:20:53.493169 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:20:53.494939 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:53.495339 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:53.495343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:53.495343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:53.495456 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:53.495716 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:53.495717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:53.495717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:53.495989 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:20:53.496052 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:53.496341 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 13:20:53.496463 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":56,"threads_started":1}
I20260925 13:20:53.496582 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:53.487573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50201 (local address 127.0.98.1:46107)
0925 13:20:53.487715 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:53.487720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:53.487747 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:20:53.487890 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:53.487892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:53.487913 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:53.487979 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:20:53.488642 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:53.489510 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:53.490595 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:53.490983 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:53.491080 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:53.493410 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:20:53.493414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:20:53.493414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:20:53.493424 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:20:53.495132 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:53.495474 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:53.495476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:53.495477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:53.495592 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:53.495773 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:53.495774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:53.495774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:53.495999 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:20:53.496074 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:53.496375 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 13:20:53.496489 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":53,"threads_started":1}
I20260925 13:20:53.496652 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:53.487465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46107 (local address 127.0.98.2:50201)
0925 13:20:53.487517 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:53.487545 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:53.487751 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:53.488026 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:53.488030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:53.488338 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:20:53.488528 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:53.488533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:53.489613 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:53.489614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:53.490515 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:53.490517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:53.491060 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:53.491060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:53.491175 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:53.491709 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:20:53.491714 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:20:53.493319 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:20:53.495199 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:53.495200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:53.495200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:53.495394 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:53.495656 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:53.495657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:53.495657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:53.495713 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:53.496066 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:20:53.496067 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:20:53.496186 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 13:20:53.496274 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 13:20:53.496538 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":150}
I20260925 13:20:53.496614 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:53.487465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42191 (local address 127.0.98.2:45811)
0925 13:20:53.487617 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:53.487627 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:53.487751 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:53.488390 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:53.488391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:53.488554 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:20:53.488612 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:53.488614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:53.489657 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:53.489658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:53.490514 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:53.490517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:53.491058 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:53.491059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:53.491174 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:53.491653 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:20:53.491661 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:20:53.493023 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:20:53.495021 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:53.495028 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:53.495030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:53.495260 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:53.495535 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:53.495536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:53.495536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:53.495658 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:53.496055 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:20:53.496062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:20:53.496183 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 13:20:53.496269 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 13:20:53.496466 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":101,"thread_start_us":44,"threads_started":1}
I20260925 13:20:53.497371 16963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6225a690eee4f54bff200c4683f813f" candidate_uuid: "171c50c01da9426585e416bc06cb627a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6f8f41766d1e44a38206e3c63e115c93" is_pre_election: true
I20260925 13:20:53.497555 16963 raft_consensus.cc:2454] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c50c01da9426585e416bc06cb627a in term 1.
W20260925 13:20:53.497705 17033 leader_election.cc:341] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a395365b68ce4658ae97b8d4d8e20e98 (127.0.98.3:42191): Not found: Tablet not found: c6225a690eee4f54bff200c4683f813f
I20260925 13:20:53.497792 17030 leader_election.cc:302] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c50c01da9426585e416bc06cb627a, 6f8f41766d1e44a38206e3c63e115c93; no voters: a395365b68ce4658ae97b8d4d8e20e98
I20260925 13:20:53.497905 17285 raft_consensus.cc:2781] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 13:20:53.497962 17285 raft_consensus.cc:482] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:20:53.497985 17285 raft_consensus.cc:3037] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 1 FOLLOWER]: Advancing to term 2
I20260925 13:20:53.499292 17285 raft_consensus.cc:504] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:53.499449 17285 leader_election.cc:288] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [CANDIDATE]: Term 2 election: Requested vote from peers a395365b68ce4658ae97b8d4d8e20e98 (127.0.98.3:42191), 6f8f41766d1e44a38206e3c63e115c93 (127.0.98.1:46107)
I20260925 13:20:53.499639 16963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6225a690eee4f54bff200c4683f813f" candidate_uuid: "171c50c01da9426585e416bc06cb627a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6f8f41766d1e44a38206e3c63e115c93"
W20260925 13:20:53.499711 17033 leader_election.cc:341] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer a395365b68ce4658ae97b8d4d8e20e98 (127.0.98.3:42191): Not found: Tablet not found: c6225a690eee4f54bff200c4683f813f
I20260925 13:20:53.499747 16963 raft_consensus.cc:3037] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 1 FOLLOWER]: Advancing to term 2
I20260925 13:20:53.501066 16963 raft_consensus.cc:2454] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c50c01da9426585e416bc06cb627a in term 2.
I20260925 13:20:53.501219 17030 leader_election.cc:302] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c50c01da9426585e416bc06cb627a, 6f8f41766d1e44a38206e3c63e115c93; no voters: a395365b68ce4658ae97b8d4d8e20e98
I20260925 13:20:53.501312 17285 raft_consensus.cc:2781] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 2 FOLLOWER]: Leader election won for term 2
I20260925 13:20:53.501446 17285 raft_consensus.cc:686] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [term 2 LEADER]: Becoming Leader. State: Replica: 171c50c01da9426585e416bc06cb627a, State: Running, Role: LEADER
I20260925 13:20:53.501559 17285 consensus_queue.cc:237] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:53.501744 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:53.501629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:51549)
0925 13:20:53.501683 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:53.501717 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 13:20:53.890592 17033 consensus_peers.cc:487] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a -> Peer a395365b68ce4658ae97b8d4d8e20e98 (127.0.98.3:42191): Couldn't send request to peer a395365b68ce4658ae97b8d4d8e20e98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6225a690eee4f54bff200c4683f813f. This is attempt 1: this message will repeat every 5th retry.
I20260925 13:20:54.048869 16963 raft_consensus.cc:1261] T c6225a690eee4f54bff200c4683f813f P 6f8f41766d1e44a38206e3c63e115c93 [term 2 FOLLOWER]: Refusing update from remote peer 171c50c01da9426585e416bc06cb627a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 13:20:54.049229 17285 consensus_queue.cc:1035] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 13:20:54.050112 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:54.049959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:58247)
0925 13:20:54.050032 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:54.050077 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60,"thread_start_us":40,"threads_started":1}
I20260925 13:20:54.050920 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:54.050776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:47801)
0925 13:20:54.050849 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:54.050880 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57,"thread_start_us":39,"threads_started":1}
I20260925 13:20:54.050966 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:54.050876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:48623)
0925 13:20:54.050912 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:54.050942 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 13:20:54.382257 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:54.382025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:59799)
0925 13:20:54.382165 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:54.382208 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260925 13:20:54.445490 17299 ts_tablet_manager.cc:927] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: Initiating tablet copy from peer 171c50c01da9426585e416bc06cb627a (127.0.98.2:42729)
I20260925 13:20:54.445756 17299 tablet_copy_client.cc:316] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:42729
Sep 25 13:20:54 dist-test-slave-t8d9 krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260925 13:20:54.453184 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:54.446170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53815 (local address 127.0.98.2:42729)
0925 13:20:54.446351 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 13:20:54.446355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:20:54.446364 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:20:54.446388 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:20:54.446390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:20:54.446403 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:20:54.446465 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:20:54.446941 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:54.447719 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:54.448447 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:20:54.448739 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:20:54.448787 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:54.450753 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:20:54.450760 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:20:54.450761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:20:54.450767 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:20:54.452185 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:54.452514 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:54.452516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:54.452516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:54.452565 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:20:54.452694 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:20:54.452695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:20:54.452695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:20:54.452826 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:20:54.452882 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:20:54.453055 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 13:20:54.453115 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":105,"threads_started":1}
I20260925 13:20:54.453202 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:54.446067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42729 (local address 127.0.98.3:53815)
0925 13:20:54.446114 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:54.446138 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 13:20:54.446218 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:20:54.446499 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:20:54.446506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:20:54.446790 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:20:54.446873 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:54.446876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:54.447798 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:54.447799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:54.448386 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:20:54.448388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:20:54.448791 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:20:54.448791 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:20:54.448864 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:20:54.449308 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:20:54.449313 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:20:54.450663 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:20:54.452261 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:54.452267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:54.452269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:54.452453 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:54.452608 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:20:54.452609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:20:54.452609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:20:54.452656 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:20:54.452895 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:20:54.452902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:20:54.452956 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 13:20:54.453028 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 13:20:54.453121 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 13:20:54.453658 17118 tablet_copy_service.cc:140] P 171c50c01da9426585e416bc06cb627a: Received BeginTabletCopySession request for tablet c6225a690eee4f54bff200c4683f813f from peer a395365b68ce4658ae97b8d4d8e20e98 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53815)
I20260925 13:20:54.453720 17118 tablet_copy_service.cc:161] P 171c50c01da9426585e416bc06cb627a: Beginning new tablet copy session on tablet c6225a690eee4f54bff200c4683f813f from peer a395365b68ce4658ae97b8d4d8e20e98 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53815: session id = a395365b68ce4658ae97b8d4d8e20e98-c6225a690eee4f54bff200c4683f813f
I20260925 13:20:54.454200 17118 tablet_copy_source_session.cc:205] T c6225a690eee4f54bff200c4683f813f P 171c50c01da9426585e416bc06cb627a: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 13:20:54.455125 17299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6225a690eee4f54bff200c4683f813f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:20:54.457867 17299 tablet_copy_client.cc:791] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: tablet copy: Starting download of 0 data blocks...
I20260925 13:20:54.458010 17299 tablet_copy_client.cc:655] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: tablet copy: Starting download of 1 WAL segments...
I20260925 13:20:54.458781 17299 tablet_copy_client.cc:531] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 13:20:54.459444 17299 tablet_bootstrap.cc:492] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: Bootstrap starting.
I20260925 13:20:54.460681 17299 log.cc:824] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: Log is configured to *not* fsync() on all Append() calls
I20260925 13:20:54.463022 17299 tablet_bootstrap.cc:492] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 13:20:54.463189 17299 tablet_bootstrap.cc:492] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: Bootstrap complete.
I20260925 13:20:54.463279 17299 ts_tablet_manager.cc:1397] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 13:20:54.464203 17299 raft_consensus.cc:348] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:54.464357 17299 raft_consensus.cc:729] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a395365b68ce4658ae97b8d4d8e20e98, State: Initialized, Role: FOLLOWER
I20260925 13:20:54.464453 17299 consensus_queue.cc:260] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "171c50c01da9426585e416bc06cb627a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42729 } } peers { permanent_uuid: "a395365b68ce4658ae97b8d4d8e20e98" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42191 } } peers { permanent_uuid: "6f8f41766d1e44a38206e3c63e115c93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46107 } }
I20260925 13:20:54.464682 17299 ts_tablet_manager.cc:1428] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20260925 13:20:54.464987 17118 tablet_copy_service.cc:342] P 171c50c01da9426585e416bc06cb627a: Request end of tablet copy session a395365b68ce4658ae97b8d4d8e20e98-c6225a690eee4f54bff200c4683f813f received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53815
I20260925 13:20:54.465049 17118 tablet_copy_service.cc:433] P 171c50c01da9426585e416bc06cb627a: ending tablet copy session a395365b68ce4658ae97b8d4d8e20e98-c6225a690eee4f54bff200c4683f813f on tablet c6225a690eee4f54bff200c4683f813f with peer a395365b68ce4658ae97b8d4d8e20e98
I20260925 13:20:54.465107 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:54.464929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:48311)
0925 13:20:54.465037 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:54.465076 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 13:20:54.465453 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:54.465363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:33435)
0925 13:20:54.465403 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:54.465430 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 13:20:54.872598 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:54.872416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:48675)
0925 13:20:54.872516 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:54.872553 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":55,"threads_started":1}
I20260925 13:20:54.872570 17232 raft_consensus.cc:1203] T c6225a690eee4f54bff200c4683f813f P a395365b68ce4658ae97b8d4d8e20e98 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260925 13:20:54.873015 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:54.872870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:38147)
0925 13:20:54.872940 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:54.872981 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 13:20:55.051585 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:55.051398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:39535)
0925 13:20:55.051500 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:55.051537 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
I20260925 13:20:55.873278 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:55.873000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:41531)
0925 13:20:55.873155 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:55.873227 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":63,"threads_started":1}
I20260925 13:20:55.873651 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:55.873447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:40231)
0925 13:20:55.873581 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:55.873612 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":57,"threads_started":1}
I20260925 13:20:56.052254 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:56.052033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:51083)
0925 13:20:56.052161 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:56.052201 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260925 13:20:56.873950 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:56.873720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:36017)
0925 13:20:56.873871 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:56.873902 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260925 13:20:56.874205 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:56.874021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:44991)
0925 13:20:56.874130 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:56.874164 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
I20260925 13:20:57.053045 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:57.052798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:59709)
0925 13:20:57.052965 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:57.052998 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260925 13:20:57.874614 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:57.874379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:59821)
0925 13:20:57.874515 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:57.874557 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260925 13:20:57.874856 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:57.874630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:42599)
0925 13:20:57.874774 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:57.874813 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260925 13:20:57.875023 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:57.874927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:54249)
0925 13:20:57.874969 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:57.874998 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 13:20:58.053745 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:58.053498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:51489)
0925 13:20:58.053662 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:58.053696 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":112,"threads_started":1}
I20260925 13:20:58.875595 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:58.875351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:43951)
0925 13:20:58.875511 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:58.875547 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
I20260925 13:20:58.875604 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:58.875340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:55431)
0925 13:20:58.875517 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:58.875563 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":90,"threads_started":1}
I20260925 13:20:59.054549 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:59.054285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:44319)
0925 13:20:59.054453 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:59.054502 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260925 13:20:59.876330 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:59.876110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:60573)
0925 13:20:59.876253 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:59.876288 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260925 13:20:59.876331 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:20:59.876110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:46405)
0925 13:20:59.876233 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 13:20:59.876275 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260925 13:21:00.055337 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:00.055096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:35323)
0925 13:21:00.055238 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:00.055282 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260925 13:21:00.877075 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:00.876854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:52889)
0925 13:21:00.876980 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:00.877021 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20260925 13:21:00.877075 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:00.876821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:51335)
0925 13:21:00.876980 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:00.877020 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260925 13:21:01.056053 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:01.055826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:48979)
0925 13:21:01.055954 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:01.056003 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260925 13:21:01.877866 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:01.877636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:53249)
0925 13:21:01.877790 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:01.877822 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":55,"threads_started":1}
I20260925 13:21:01.877866 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:01.877629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:49945)
0925 13:21:01.877757 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:01.877802 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260925 13:21:02.056862 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:02.056548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:53151)
0925 13:21:02.056713 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:02.056809 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260925 13:21:02.878649 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:02.878411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:49643)
0925 13:21:02.878551 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:02.878595 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260925 13:21:02.878650 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:02.878406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:55131)
0925 13:21:02.878569 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:02.878601 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
I20260925 13:21:03.057646 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:03.057410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:57035)
0925 13:21:03.057547 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:03.057595 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260925 13:21:03.879386 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:03.879150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:41109)
0925 13:21:03.879283 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:03.879329 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260925 13:21:03.879386 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:03.879154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:54901)
0925 13:21:03.879312 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:03.879346 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":69,"threads_started":1}
I20260925 13:21:04.058326 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:04.058089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:53969)
0925 13:21:04.058224 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:04.058268 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260925 13:21:04.880127 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:04.879917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:60095)
0925 13:21:04.880045 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:04.880081 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260925 13:21:04.880127 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:04.879889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:53571)
0925 13:21:04.880038 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:04.880083 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260925 13:21:05.059010 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:05.058769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:42849)
0925 13:21:05.058918 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:05.058963 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260925 13:21:05.880828 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:05.880598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:41141)
0925 13:21:05.880734 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:05.880774 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260925 13:21:05.880854 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:05.880598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:52969)
0925 13:21:05.880763 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:05.880797 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
I20260925 13:21:06.059803 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:06.059542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:46833)
0925 13:21:06.059705 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:06.059754 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260925 13:21:06.881618 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:06.881372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.2:55957)
0925 13:21:06.881513 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:06.881566 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260925 13:21:06.881618 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:06.881372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.3:55627)
0925 13:21:06.881503 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:06.881567 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260925 13:21:07.060648 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:07.060425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34629 (local address 127.0.98.1:58433)
0925 13:21:07.060556 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:07.060600 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260925 13:21:07.389034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 16845
I20260925 13:21:07.393117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 17013
I20260925 13:21:07.396984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 17148
2026-09-25T13:21:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58052 ms)
[----------] 30 tests from SecurityITest (227567 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:07 dist-test-slave-t8d9 krb5kdc[17348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:07 dist-test-slave-t8d9 krb5kdc[17348](info): set up 2 sockets
Sep 25 13:21:07 dist-test-slave-t8d9 krb5kdc[17348](info): commencing operation
krb5kdc: starting...
W20260925 13:21:08.349085 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:08 dist-test-slave-t8d9 krb5kdc[17348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:21:08.508627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33255
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:08.588820 17361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:08.589047 17361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:08.589071 17361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:08.590267 17361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:08.590312 17361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:08.590325 17361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:08.590337 17361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:08.590346 17361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:08.591836 17361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:08.592048 17361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:08.592224 17361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:08.596848 17367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:08.596869 17366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:08.596901 17361 server_base.cc:1034] running on GCE node
W20260925 13:21:08.596869 17369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:08.597272 17361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:08.597564 17361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:08.598707 17361 hybrid_clock.cc:648] HybridClock initialized: now 1790342468598699 us; error 30 us; skew 500 ppm
Sep 25 13:21:08 dist-test-slave-t8d9 krb5kdc[17348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:08.601902 17361 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:21:08.602474 17361 webserver.cc:460] Webserver started at http://127.0.98.62:34229/ using document root <none> and password file <none>
I20260925 13:21:08.602695 17361 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:08.602739 17361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:08.602874 17361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:08.603951 17361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "acb169a152bc490495f3e98cd255e6c0"
format_stamp: "Formatted at 2026-09-25 13:21:08 on dist-test-slave-t8d9"
server_key: "d062d90d76500ce30a730d615558379f"
server_key_iv: "bffaa0f3df1f4f36a2922d3dc95a711a"
server_key_version: "encryptionkey@0"
I20260925 13:21:08.604290 17361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "acb169a152bc490495f3e98cd255e6c0"
format_stamp: "Formatted at 2026-09-25 13:21:08 on dist-test-slave-t8d9"
server_key: "d062d90d76500ce30a730d615558379f"
server_key_iv: "bffaa0f3df1f4f36a2922d3dc95a711a"
server_key_version: "encryptionkey@0"
I20260925 13:21:08.605625 17361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:21:08.606382 17376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:08.606513 17361 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:08.606598 17361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "acb169a152bc490495f3e98cd255e6c0"
format_stamp: "Formatted at 2026-09-25 13:21:08 on dist-test-slave-t8d9"
server_key: "d062d90d76500ce30a730d615558379f"
server_key_iv: "bffaa0f3df1f4f36a2922d3dc95a711a"
server_key_version: "encryptionkey@0"
I20260925 13:21:08.606668 17361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:08.640408 17361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:08.642521 17361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:08.642645 17361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:08.646503 17361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33255
I20260925 13:21:08.646543 17428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33255 every 8 connection(s)
I20260925 13:21:08.646885 17361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:08.647727 17429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:08.650121 17429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0: Bootstrap starting.
I20260925 13:21:08.650689 17429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:08.650951 17429 log.cc:824] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:08.651620 17429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0: No bootstrap required, opened a new log
I20260925 13:21:08.653323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 17361
I20260925 13:21:08.653204 17429 raft_consensus.cc:348] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb169a152bc490495f3e98cd255e6c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33255 } }
I20260925 13:21:08.653398 17429 raft_consensus.cc:374] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:08.653439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:08.653432 17429 raft_consensus.cc:729] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb169a152bc490495f3e98cd255e6c0, State: Initialized, Role: FOLLOWER
I20260925 13:21:08.653657 392 external_mini_cluster.cc:1428] Setting key fa48f3275c7a26c92059274b7f721db5
I20260925 13:21:08.653618 17429 consensus_queue.cc:260] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb169a152bc490495f3e98cd255e6c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33255 } }
I20260925 13:21:08.653697 17429 raft_consensus.cc:388] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:08.653726 17429 raft_consensus.cc:482] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:08.653776 17429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:08.654417 17429 raft_consensus.cc:504] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb169a152bc490495f3e98cd255e6c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33255 } }
I20260925 13:21:08.654531 17429 leader_election.cc:302] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acb169a152bc490495f3e98cd255e6c0; no voters:
I20260925 13:21:08.654676 17429 leader_election.cc:288] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:08.654706 17434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:08.654835 17434 raft_consensus.cc:686] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [term 1 LEADER]: Becoming Leader. State: Replica: acb169a152bc490495f3e98cd255e6c0, State: Running, Role: LEADER
I20260925 13:21:08.654933 17429 sys_catalog.cc:564] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:08.654978 17434 consensus_queue.cc:237] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb169a152bc490495f3e98cd255e6c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33255 } }
I20260925 13:21:08.655344 17435 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acb169a152bc490495f3e98cd255e6c0. Latest consensus state: current_term: 1 leader_uuid: "acb169a152bc490495f3e98cd255e6c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb169a152bc490495f3e98cd255e6c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33255 } } }
I20260925 13:21:08.655414 17435 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:08.655649 17436 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acb169a152bc490495f3e98cd255e6c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb169a152bc490495f3e98cd255e6c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33255 } } }
I20260925 13:21:08.655961 17436 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:08.656088 17441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:08.656625 17441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:08.658197 17441 catalog_manager.cc:1347] Generated new cluster ID: 6548ddc3031a4c86b2012ff988f1ab15
I20260925 13:21:08.658252 17441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:21:08 dist-test-slave-t8d9 krb5kdc[17348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:08.664283 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:08.654504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.0.98.62:33255)
0925 13:21:08.654702 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:08.654713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:08.654755 (+ 42us) server_negotiation.cc:407] Connection header received
0925 13:21:08.654883 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:08.654890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:08.654952 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:08.655922 (+ 970us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:08.656061 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:08.657524 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:08.658434 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:08.658894 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:08.658991 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:08.661197 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:08.661201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:08.661202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:08.661215 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:08.662850 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:08.663173 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:08.663177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:08.663177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:08.663268 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:08.663445 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:08.663446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:08.663446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:08.663696 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:08.663877 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:08.663947 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 13:21:08.664089 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260925 13:21:08.681974 17441 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:08.682716 17441 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:08.687135 17441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0: Generated new TSK 0
I20260925 13:21:08.687335 17441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:21:08.729379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33255
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:08.811223 17455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:08.811447 17455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:08.811482 17455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:08.812847 17455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:08.812935 17455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:08.812981 17455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:21:08.814647 17455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:08.814901 17455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:08.815130 17455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:08.819985 17460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:08.820114 17455 server_base.cc:1034] running on GCE node
W20260925 13:21:08.819999 17463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:08.820196 17461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:08.820482 17455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:08.820716 17455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:08.821852 17455 hybrid_clock.cc:648] HybridClock initialized: now 1790342468821845 us; error 33 us; skew 500 ppm
Sep 25 13:21:08 dist-test-slave-t8d9 krb5kdc[17348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:08.824312 17455 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:21:08.824743 17455 webserver.cc:460] Webserver started at http://127.0.98.1:36797/ using document root <none> and password file <none>
I20260925 13:21:08.824939 17455 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:08.824978 17455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:08.825071 17455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:08.826119 17455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "633b9cffbaf046299be77b1df5bcdab8"
format_stamp: "Formatted at 2026-09-25 13:21:08 on dist-test-slave-t8d9"
server_key: "0273e3f1d8208e41930c30b2cd77df37"
server_key_iv: "b01e277359ae58f2eeb8e7612cee6110"
server_key_version: "encryptionkey@0"
I20260925 13:21:08.826404 17455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "633b9cffbaf046299be77b1df5bcdab8"
format_stamp: "Formatted at 2026-09-25 13:21:08 on dist-test-slave-t8d9"
server_key: "0273e3f1d8208e41930c30b2cd77df37"
server_key_iv: "b01e277359ae58f2eeb8e7612cee6110"
server_key_version: "encryptionkey@0"
I20260925 13:21:08.827555 17455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:21:08.828176 17470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:08.828322 17455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:08.828402 17455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "633b9cffbaf046299be77b1df5bcdab8"
format_stamp: "Formatted at 2026-09-25 13:21:08 on dist-test-slave-t8d9"
server_key: "0273e3f1d8208e41930c30b2cd77df37"
server_key_iv: "b01e277359ae58f2eeb8e7612cee6110"
server_key_version: "encryptionkey@0"
I20260925 13:21:08.828469 17455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:08.846151 17455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:08.848407 17455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:08.848567 17455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:08.848789 17455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:08.849094 17455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:08.849128 17455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:08.849156 17455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:08.849174 17455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:08.854666 17455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38687
I20260925 13:21:08.854719 17583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38687 every 8 connection(s)
I20260925 13:21:08.855033 17455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 13:21:08 dist-test-slave-t8d9 krb5kdc[17348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:08.864076 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:08.855794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47279 (local address 127.0.98.62:33255)
0925 13:21:08.855840 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:08.855842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:08.856045 (+ 203us) server_negotiation.cc:407] Connection header received
0925 13:21:08.856989 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:08.856990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:08.857005 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:08.857053 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:08.857673 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:08.858164 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:08.859158 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:08.859341 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:08.859382 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:08.861474 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:08.861475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:08.861476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:08.861484 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:08.862700 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:08.863140 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:08.863141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:08.863142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:08.863208 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:08.863453 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:08.863454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:08.863454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:08.863585 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:08.863649 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:08.863929 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 13:21:08.864007 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:21:08.864200 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:08.855776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33255 (local address 127.0.98.1:47279)
0925 13:21:08.855926 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:08.855964 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:08.856899 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:08.857115 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:08.857120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:08.857377 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:08.857607 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:08.857613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:08.858243 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:08.858244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:08.859094 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:08.859096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:08.859395 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:08.859395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:08.859506 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:08.860019 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:08.860028 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:08.861362 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:08.862792 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:08.862799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:08.862801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:08.863047 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:08.863268 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:08.863269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:08.863270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:08.863375 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:08.863642 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:08.863645 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:08.863769 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 13:21:08.863850 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 13:21:08.863992 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260925 13:21:08.864605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 17455
I20260925 13:21:08.864709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:21:08.864795 17584 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33255
I20260925 13:21:08.864892 17584 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:08.864912 392 external_mini_cluster.cc:1428] Setting key 2859c9dbf20aa46bb9261a98e75df51d
I20260925 13:21:08.865087 17584 heartbeater.cc:511] Master 127.0.98.62:33255 requested a full tablet report, sending...
I20260925 13:21:08.865710 17392 ts_manager.cc:194] Registered new tserver with Master: 633b9cffbaf046299be77b1df5bcdab8 (127.0.98.1:38687)
I20260925 13:21:08.866308 17392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47279
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:21:08.889415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33255
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:08.969085 17589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:08.969307 17589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:08.969341 17589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:08.970561 17589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:08.970600 17589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:08.970634 17589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:21:08.972189 17589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:08.972434 17589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:08.972653 17589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:08.977445 17597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:08.977442 17595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:08.977592 17589 server_base.cc:1034] running on GCE node
W20260925 13:21:08.977445 17594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:08.977871 17589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:08.978116 17589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:08.979260 17589 hybrid_clock.cc:648] HybridClock initialized: now 1790342468979246 us; error 30 us; skew 500 ppm
Sep 25 13:21:08 dist-test-slave-t8d9 krb5kdc[17348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:08.981737 17589 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:21:08.982133 17589 webserver.cc:460] Webserver started at http://127.0.98.2:42895/ using document root <none> and password file <none>
I20260925 13:21:08.982314 17589 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:08.982352 17589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:08.982448 17589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:08.983418 17589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "ca69a30e07224fc39e31d1a89a4d16ea"
format_stamp: "Formatted at 2026-09-25 13:21:08 on dist-test-slave-t8d9"
server_key: "4b056da0ec875481298962dcbfbc61bb"
server_key_iv: "0cfb56692aba9a5b5f40476f5a885031"
server_key_version: "encryptionkey@0"
I20260925 13:21:08.983716 17589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ca69a30e07224fc39e31d1a89a4d16ea"
format_stamp: "Formatted at 2026-09-25 13:21:08 on dist-test-slave-t8d9"
server_key: "4b056da0ec875481298962dcbfbc61bb"
server_key_iv: "0cfb56692aba9a5b5f40476f5a885031"
server_key_version: "encryptionkey@0"
I20260925 13:21:08.984920 17589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:21:08.985556 17604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:08.985697 17589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:08.985772 17589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "ca69a30e07224fc39e31d1a89a4d16ea"
format_stamp: "Formatted at 2026-09-25 13:21:08 on dist-test-slave-t8d9"
server_key: "4b056da0ec875481298962dcbfbc61bb"
server_key_iv: "0cfb56692aba9a5b5f40476f5a885031"
server_key_version: "encryptionkey@0"
I20260925 13:21:08.985826 17589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:09.006114 17589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:09.008010 17589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:09.008133 17589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:09.008324 17589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:09.008641 17589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:09.008675 17589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:09.008697 17589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:09.008713 17589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:09.014137 17589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46877
I20260925 13:21:09.014189 17717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46877 every 8 connection(s)
I20260925 13:21:09.014556 17589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 13:21:09 dist-test-slave-t8d9 krb5kdc[17348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:09.023298 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.015236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34473 (local address 127.0.98.62:33255)
0925 13:21:09.015300 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:09.015302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:09.015522 (+ 220us) server_negotiation.cc:407] Connection header received
0925 13:21:09.016456 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:09.016458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:09.016470 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:09.016525 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:09.017118 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.017645 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.018620 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.018802 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.018849 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.021008 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:09.021010 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:09.021010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:09.021017 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:09.022204 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:09.022568 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:09.022568 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:09.022569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:09.022613 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:09.022791 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:09.022791 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:09.022791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:09.022893 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:09.022937 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:09.023156 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 13:21:09.023229 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:21:09.023468 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.015212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33255 (local address 127.0.98.2:34473)
0925 13:21:09.015381 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:09.015425 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:09.016349 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:09.016581 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:09.016585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:09.016850 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:09.017046 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.017050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.017721 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.017722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.018556 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.018558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.018857 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.018858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.018956 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.019472 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:09.019481 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:09.020916 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:09.022271 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:09.022277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:09.022279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:09.022511 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:09.022657 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:09.022657 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:09.022657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:09.022751 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:09.022933 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:09.022936 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:09.023059 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 13:21:09.023137 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 13:21:09.023263 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260925 13:21:09.023928 17718 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33255
I20260925 13:21:09.024024 17718 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:09.024178 17718 heartbeater.cc:511] Master 127.0.98.62:33255 requested a full tablet report, sending...
I20260925 13:21:09.024428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 17589
I20260925 13:21:09.024528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:21:09.024616 17392 ts_manager.cc:194] Registered new tserver with Master: ca69a30e07224fc39e31d1a89a4d16ea (127.0.98.2:46877)
I20260925 13:21:09.024736 392 external_mini_cluster.cc:1428] Setting key 612f478ac6ad7eab03a348f695964b91
I20260925 13:21:09.025079 17392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34473
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:21:09.061756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33255
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:09.142596 17723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:09.142822 17723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:09.142853 17723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:09.144016 17723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:09.144059 17723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:09.144093 17723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:21:09.145650 17723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:09.145905 17723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:09.146121 17723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:09.150789 17731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:09.150831 17729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:09.150831 17723 server_base.cc:1034] running on GCE node
W20260925 13:21:09.150805 17728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:09.151170 17723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:09.151404 17723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:09.152539 17723 hybrid_clock.cc:648] HybridClock initialized: now 1790342469152528 us; error 29 us; skew 500 ppm
Sep 25 13:21:09 dist-test-slave-t8d9 krb5kdc[17348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:09.155014 17723 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:21:09.155476 17723 webserver.cc:460] Webserver started at http://127.0.98.3:40477/ using document root <none> and password file <none>
I20260925 13:21:09.155687 17723 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:09.155733 17723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:09.155856 17723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:09.156976 17723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "f122179d4bb342aa94a59eea35597b91"
format_stamp: "Formatted at 2026-09-25 13:21:09 on dist-test-slave-t8d9"
server_key: "dbc2f72ad806021eec27a55b5adc75c1"
server_key_iv: "3ef08ff8ed5693b838a3989855fac9ea"
server_key_version: "encryptionkey@0"
I20260925 13:21:09.157287 17723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f122179d4bb342aa94a59eea35597b91"
format_stamp: "Formatted at 2026-09-25 13:21:09 on dist-test-slave-t8d9"
server_key: "dbc2f72ad806021eec27a55b5adc75c1"
server_key_iv: "3ef08ff8ed5693b838a3989855fac9ea"
server_key_version: "encryptionkey@0"
I20260925 13:21:09.158520 17723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:21:09.159175 17738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:09.159322 17723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:09.159400 17723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "f122179d4bb342aa94a59eea35597b91"
format_stamp: "Formatted at 2026-09-25 13:21:09 on dist-test-slave-t8d9"
server_key: "dbc2f72ad806021eec27a55b5adc75c1"
server_key_iv: "3ef08ff8ed5693b838a3989855fac9ea"
server_key_version: "encryptionkey@0"
I20260925 13:21:09.159467 17723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:09.171766 17723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:09.173998 17723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:09.174146 17723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:09.174353 17723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:09.174606 17723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:09.174635 17723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:09.174655 17723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:09.174671 17723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:09.180058 17723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45409
I20260925 13:21:09.180110 17851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45409 every 8 connection(s)
I20260925 13:21:09.180415 17723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 13:21:09 dist-test-slave-t8d9 krb5kdc[17348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:09.186340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 17723
I20260925 13:21:09.186424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:21:09.186581 392 external_mini_cluster.cc:1428] Setting key f1e8dd00f22c2834c60d8f7170f65feb
I20260925 13:21:09.189488 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.181081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52397 (local address 127.0.98.62:33255)
0925 13:21:09.181140 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:09.181142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:09.181327 (+ 185us) server_negotiation.cc:407] Connection header received
0925 13:21:09.182236 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:09.182237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:09.182246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:09.182303 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:09.182953 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.183448 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.184430 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.184640 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.184695 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.186844 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:09.186846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:09.186846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:09.186854 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:09.188316 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:09.188656 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:09.188657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:09.188657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:09.188707 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:09.188888 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:09.188888 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:09.188889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:09.188992 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:09.189042 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:09.189300 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 13:21:09.189372 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:21:09.189636 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.181051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33255 (local address 127.0.98.3:52397)
0925 13:21:09.181206 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:09.181247 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:09.182152 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:09.182380 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:09.182383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:09.182640 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:09.182835 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.182840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.183521 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.183521 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.184345 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.184347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.184706 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.184707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.184815 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.185277 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:09.185285 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:09.186760 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:09.188382 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:09.188388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:09.188390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:09.188602 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:09.188741 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:09.188742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:09.188742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:09.188844 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:09.189040 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:09.189042 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:09.189156 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 13:21:09.189238 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 13:21:09.189406 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260925 13:21:09.190090 17852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33255
I20260925 13:21:09.190174 17852 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:09.190335 17852 heartbeater.cc:511] Master 127.0.98.62:33255 requested a full tablet report, sending...
I20260925 13:21:09.190644 17393 ts_manager.cc:194] Registered new tserver with Master: f122179d4bb342aa94a59eea35597b91 (127.0.98.3:45409)
I20260925 13:21:09.190989 17393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52397
I20260925 13:21:09.198251 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:21:09.204866 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.199505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.0.98.62:33255)
0925 13:21:09.199570 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:09.199573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:09.199655 (+ 82us) server_negotiation.cc:407] Connection header received
0925 13:21:09.199717 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:09.199718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:09.199728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:09.199774 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:09.200298 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.200805 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.201507 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.201758 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.201814 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.202847 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:09.202848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:09.202849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:09.202856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:09.203864 (+ 1008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:09.204153 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:09.204154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:09.204154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:09.204207 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:09.204359 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:09.204360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:09.204360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:09.204513 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:09.204599 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:09.204702 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 13:21:09.204776 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:21:09.206394 17393 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:21:09.208103 17393 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:21:09.214504 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.210298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38687 (local address 127.0.0.1:56204)
0925 13:21:09.210796 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:09.210804 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:09.210849 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:09.211014 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:09.211014 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:09.211015 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:09.211051 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.211052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.212310 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.212311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.213363 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.213365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.214178 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.214179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.214264 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.214278 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 13:21:09.214343 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 13:21:09.214401 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":77,"threads_started":1}
I20260925 13:21:09.214490 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.210419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.0.98.1:38687)
0925 13:21:09.210606 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:09.210612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:09.210850 (+ 238us) server_negotiation.cc:407] Connection header received
0925 13:21:09.210908 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:09.210912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:09.210926 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:09.211014 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:09.211162 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.212210 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.213583 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.214102 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.214167 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.214230 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:09.214362 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 13:21:09.214406 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260925 13:21:09.214666 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.210501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46877 (local address 127.0.0.1:34410)
0925 13:21:09.211087 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:09.211097 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:09.211152 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:09.211460 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:09.211460 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:09.211462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:09.211499 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.211501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.212615 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.212616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.213524 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.213527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.214454 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.214455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.214507 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.214510 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:09.214548 (+ 38us) client_negotiation.cc:240] Negotiation successful
0925 13:21:09.214582 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":68,"threads_started":1}
I20260925 13:21:09.214958 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.210627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.0.98.2:46877)
0925 13:21:09.210801 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:09.210805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:09.211257 (+ 452us) server_negotiation.cc:407] Connection header received
0925 13:21:09.211298 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:09.211300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:09.211315 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:09.211382 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:09.211575 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.212531 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.213628 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.214321 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.214375 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.214426 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:09.214825 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 13:21:09.214870 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260925 13:21:09.215073 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.210641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58004 (local address 127.0.98.3:45409)
0925 13:21:09.210814 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:09.210817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:09.210827 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:21:09.210868 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:09.210870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:09.210887 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:09.210972 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:09.211134 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.211994 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.213455 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.214202 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.214272 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.214331 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:09.214943 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 13:21:09.214993 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260925 13:21:09.215214 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.210291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45409 (local address 127.0.0.1:58004)
0925 13:21:09.210602 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:09.210644 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:09.210714 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:09.210974 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:09.210977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:09.210978 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:09.211050 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.211052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.212080 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.212081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.213354 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.213357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.214367 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.214368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.214458 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.214464 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 13:21:09.214508 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 13:21:09.215147 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":64,"threads_started":1}
I20260925 13:21:09.215216 17518 tablet_service.cc:1467] Processing CreateTablet for tablet cd24b97aa5374292858ac71fa6bc7aed (DEFAULT_TABLE table=test-table [id=81bc0bf2d338471d9f58a5323a1fe04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:09.215624 17518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd24b97aa5374292858ac71fa6bc7aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:09.215651 17652 tablet_service.cc:1467] Processing CreateTablet for tablet cd24b97aa5374292858ac71fa6bc7aed (DEFAULT_TABLE table=test-table [id=81bc0bf2d338471d9f58a5323a1fe04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:09.215916 17652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd24b97aa5374292858ac71fa6bc7aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:09.215947 17786 tablet_service.cc:1467] Processing CreateTablet for tablet cd24b97aa5374292858ac71fa6bc7aed (DEFAULT_TABLE table=test-table [id=81bc0bf2d338471d9f58a5323a1fe04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:09.216254 17786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd24b97aa5374292858ac71fa6bc7aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:09.218650 17869 tablet_bootstrap.cc:492] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea: Bootstrap starting.
I20260925 13:21:09.219002 17870 tablet_bootstrap.cc:492] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8: Bootstrap starting.
I20260925 13:21:09.219334 17871 tablet_bootstrap.cc:492] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91: Bootstrap starting.
I20260925 13:21:09.219523 17869 tablet_bootstrap.cc:654] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:09.219597 17870 tablet_bootstrap.cc:654] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:09.219980 17871 tablet_bootstrap.cc:654] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:09.220104 17869 log.cc:824] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:09.220155 17870 log.cc:824] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:09.220520 17871 log.cc:824] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:09.220955 17870 tablet_bootstrap.cc:492] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8: No bootstrap required, opened a new log
I20260925 13:21:09.221040 17870 ts_tablet_manager.cc:1397] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:21:09.221285 17871 tablet_bootstrap.cc:492] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91: No bootstrap required, opened a new log
I20260925 13:21:09.221362 17871 ts_tablet_manager.cc:1397] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:21:09.223155 17870 raft_consensus.cc:348] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633b9cffbaf046299be77b1df5bcdab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38687 } } peers { permanent_uuid: "f122179d4bb342aa94a59eea35597b91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } } peers { permanent_uuid: "ca69a30e07224fc39e31d1a89a4d16ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } }
I20260925 13:21:09.223371 17870 raft_consensus.cc:374] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:09.223428 17870 raft_consensus.cc:729] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633b9cffbaf046299be77b1df5bcdab8, State: Initialized, Role: FOLLOWER
I20260925 13:21:09.223415 17871 raft_consensus.cc:348] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633b9cffbaf046299be77b1df5bcdab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38687 } } peers { permanent_uuid: "f122179d4bb342aa94a59eea35597b91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } } peers { permanent_uuid: "ca69a30e07224fc39e31d1a89a4d16ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } }
I20260925 13:21:09.223591 17871 raft_consensus.cc:374] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:09.223563 17870 consensus_queue.cc:260] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633b9cffbaf046299be77b1df5bcdab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38687 } } peers { permanent_uuid: "f122179d4bb342aa94a59eea35597b91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } } peers { permanent_uuid: "ca69a30e07224fc39e31d1a89a4d16ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } }
I20260925 13:21:09.223639 17871 raft_consensus.cc:729] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f122179d4bb342aa94a59eea35597b91, State: Initialized, Role: FOLLOWER
I20260925 13:21:09.223754 17871 consensus_queue.cc:260] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633b9cffbaf046299be77b1df5bcdab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38687 } } peers { permanent_uuid: "f122179d4bb342aa94a59eea35597b91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } } peers { permanent_uuid: "ca69a30e07224fc39e31d1a89a4d16ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } }
I20260925 13:21:09.223901 17584 heartbeater.cc:503] Master 127.0.98.62:33255 was elected leader, sending a full tablet report...
I20260925 13:21:09.224032 17871 ts_tablet_manager.cc:1428] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:21:09.224054 17852 heartbeater.cc:503] Master 127.0.98.62:33255 was elected leader, sending a full tablet report...
I20260925 13:21:09.223922 17870 ts_tablet_manager.cc:1428] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:21:09.224637 17869 tablet_bootstrap.cc:492] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea: No bootstrap required, opened a new log
I20260925 13:21:09.224716 17869 ts_tablet_manager.cc:1397] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 13:21:09.226473 17869 raft_consensus.cc:348] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633b9cffbaf046299be77b1df5bcdab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38687 } } peers { permanent_uuid: "f122179d4bb342aa94a59eea35597b91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } } peers { permanent_uuid: "ca69a30e07224fc39e31d1a89a4d16ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } }
I20260925 13:21:09.226635 17869 raft_consensus.cc:374] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:09.226671 17869 raft_consensus.cc:729] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca69a30e07224fc39e31d1a89a4d16ea, State: Initialized, Role: FOLLOWER
I20260925 13:21:09.226799 17869 consensus_queue.cc:260] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633b9cffbaf046299be77b1df5bcdab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38687 } } peers { permanent_uuid: "f122179d4bb342aa94a59eea35597b91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } } peers { permanent_uuid: "ca69a30e07224fc39e31d1a89a4d16ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } }
I20260925 13:21:09.227093 17869 ts_tablet_manager.cc:1428] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 13:21:09.227121 17718 heartbeater.cc:503] Master 127.0.98.62:33255 was elected leader, sending a full tablet report...
W20260925 13:21:09.265039 17719 tablet.cc:2369] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 13:21:09.355834 17585 tablet.cc:2369] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:21:09.394356 17875 raft_consensus.cc:482] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:21:09.394483 17875 raft_consensus.cc:504] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633b9cffbaf046299be77b1df5bcdab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38687 } } peers { permanent_uuid: "f122179d4bb342aa94a59eea35597b91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } } peers { permanent_uuid: "ca69a30e07224fc39e31d1a89a4d16ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } }
I20260925 13:21:09.394797 17875 leader_election.cc:288] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f122179d4bb342aa94a59eea35597b91 (127.0.98.3:45409), ca69a30e07224fc39e31d1a89a4d16ea (127.0.98.2:46877)
I20260925 13:21:09.398698 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.395053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51317 (local address 127.0.98.3:45409)
0925 13:21:09.395103 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:09.395104 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:09.395188 (+ 84us) server_negotiation.cc:407] Connection header received
0925 13:21:09.395241 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:09.395242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:09.395251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:09.395333 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:09.395526 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.396255 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.397730 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.398310 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.398384 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.398395 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:09.398583 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 13:21:09.398626 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:21:09.398698 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.395026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52653 (local address 127.0.98.2:46877)
0925 13:21:09.395075 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:09.395077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:09.395172 (+ 95us) server_negotiation.cc:407] Connection header received
0925 13:21:09.395238 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:09.395239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:09.395251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:09.395325 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:09.395513 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.396276 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.397730 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.398311 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.398384 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.398395 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:09.398583 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 13:21:09.398621 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:21:09.398741 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.394958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45409 (local address 127.0.98.1:51317)
0925 13:21:09.395127 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:09.395135 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:09.395184 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:09.395337 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:09.395338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:09.395340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:09.395454 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.395455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.396331 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.396331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.397655 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.397656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.398389 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.398391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.398519 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.398524 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:21:09.398576 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 13:21:09.398636 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":59,"spinlock_wait_cycles":14592,"thread_start_us":78,"threads_started":1}
I20260925 13:21:09.398741 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.394958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46877 (local address 127.0.98.1:52653)
0925 13:21:09.395099 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:09.395111 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:09.395180 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:09.395337 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:09.395338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:09.395341 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:09.395443 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.395445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.396340 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.396341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.397643 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.397648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.398391 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.398392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.398519 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.398525 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 13:21:09.398576 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 13:21:09.398636 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":123,"thread_start_us":49,"threads_started":1}
I20260925 13:21:09.399164 17806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd24b97aa5374292858ac71fa6bc7aed" candidate_uuid: "633b9cffbaf046299be77b1df5bcdab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f122179d4bb342aa94a59eea35597b91" is_pre_election: true
I20260925 13:21:09.399219 17672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd24b97aa5374292858ac71fa6bc7aed" candidate_uuid: "633b9cffbaf046299be77b1df5bcdab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca69a30e07224fc39e31d1a89a4d16ea" is_pre_election: true
I20260925 13:21:09.399317 17806 raft_consensus.cc:2454] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633b9cffbaf046299be77b1df5bcdab8 in term 0.
I20260925 13:21:09.399375 17672 raft_consensus.cc:2454] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633b9cffbaf046299be77b1df5bcdab8 in term 0.
I20260925 13:21:09.399580 17471 leader_election.cc:302] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 633b9cffbaf046299be77b1df5bcdab8, f122179d4bb342aa94a59eea35597b91; no voters:
I20260925 13:21:09.399721 17875 raft_consensus.cc:2781] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:21:09.399786 17875 raft_consensus.cc:482] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:21:09.399812 17875 raft_consensus.cc:3037] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:09.400513 17875 raft_consensus.cc:504] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633b9cffbaf046299be77b1df5bcdab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38687 } } peers { permanent_uuid: "f122179d4bb342aa94a59eea35597b91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } } peers { permanent_uuid: "ca69a30e07224fc39e31d1a89a4d16ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } }
I20260925 13:21:09.400630 17875 leader_election.cc:288] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [CANDIDATE]: Term 1 election: Requested vote from peers f122179d4bb342aa94a59eea35597b91 (127.0.98.3:45409), ca69a30e07224fc39e31d1a89a4d16ea (127.0.98.2:46877)
I20260925 13:21:09.400787 17672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd24b97aa5374292858ac71fa6bc7aed" candidate_uuid: "633b9cffbaf046299be77b1df5bcdab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca69a30e07224fc39e31d1a89a4d16ea"
I20260925 13:21:09.400802 17806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd24b97aa5374292858ac71fa6bc7aed" candidate_uuid: "633b9cffbaf046299be77b1df5bcdab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f122179d4bb342aa94a59eea35597b91"
I20260925 13:21:09.400861 17672 raft_consensus.cc:3037] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:09.400877 17806 raft_consensus.cc:3037] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:09.401526 17806 raft_consensus.cc:2454] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633b9cffbaf046299be77b1df5bcdab8 in term 1.
I20260925 13:21:09.401609 17672 raft_consensus.cc:2454] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633b9cffbaf046299be77b1df5bcdab8 in term 1.
I20260925 13:21:09.401732 17471 leader_election.cc:302] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 633b9cffbaf046299be77b1df5bcdab8, f122179d4bb342aa94a59eea35597b91; no voters:
I20260925 13:21:09.401829 17875 raft_consensus.cc:2781] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:09.401997 17875 raft_consensus.cc:686] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [term 1 LEADER]: Becoming Leader. State: Replica: 633b9cffbaf046299be77b1df5bcdab8, State: Running, Role: LEADER
I20260925 13:21:09.402091 17875 consensus_queue.cc:237] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633b9cffbaf046299be77b1df5bcdab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38687 } } peers { permanent_uuid: "f122179d4bb342aa94a59eea35597b91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } } peers { permanent_uuid: "ca69a30e07224fc39e31d1a89a4d16ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } }
I20260925 13:21:09.402932 17393 catalog_manager.cc:5495] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 633b9cffbaf046299be77b1df5bcdab8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "633b9cffbaf046299be77b1df5bcdab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633b9cffbaf046299be77b1df5bcdab8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f122179d4bb342aa94a59eea35597b91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca69a30e07224fc39e31d1a89a4d16ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 } health_report { overall_health: UNKNOWN } } }
W20260925 13:21:09.431017 17853 tablet.cc:2369] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:21:09.439065 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.436917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.0.98.1:38687)
0925 13:21:09.436989 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:09.436991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:09.436999 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:21:09.437037 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:09.437038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:09.437058 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:09.437120 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:09.437219 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.437718 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.438400 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.438577 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.438618 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.438761 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:09.438880 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:09.438953 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:09.438965 (+ 12us) server_negotiation.cc:299] Negotiation successful
0925 13:21:09.438984 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 13:21:09.440932 17806 raft_consensus.cc:1261] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91 [term 1 FOLLOWER]: Refusing update from remote peer 633b9cffbaf046299be77b1df5bcdab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:09.440930 17672 raft_consensus.cc:1261] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea [term 1 FOLLOWER]: Refusing update from remote peer 633b9cffbaf046299be77b1df5bcdab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:09.441305 17880 consensus_queue.cc:1035] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f122179d4bb342aa94a59eea35597b91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:21:09.441442 17875 consensus_queue.cc:1035] T cd24b97aa5374292858ac71fa6bc7aed P 633b9cffbaf046299be77b1df5bcdab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca69a30e07224fc39e31d1a89a4d16ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:21:09.443892 17884 mvcc.cc:205] Tried to move back new op lower bound from 7333242754827206656 to 7333242754671566848. Current Snapshot: MvccSnapshot[applied={T|T < 7333242754827206656}]
I20260925 13:21:09.446111 17393 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:21:09.446192 17393 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:21:09.446895 17393 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acb169a152bc490495f3e98cd255e6c0: Sending DeleteTablet for 3 replicas of tablet cd24b97aa5374292858ac71fa6bc7aed
I20260925 13:21:09.447188 17518 tablet_service.cc:1514] Processing DeleteTablet for tablet cd24b97aa5374292858ac71fa6bc7aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56204
I20260925 13:21:09.447254 17652 tablet_service.cc:1514] Processing DeleteTablet for tablet cd24b97aa5374292858ac71fa6bc7aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34410
I20260925 13:21:09.447292 17786 tablet_service.cc:1514] Processing DeleteTablet for tablet cd24b97aa5374292858ac71fa6bc7aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58004
I20260925 13:21:09.447490 17894 tablet_replica.cc:330] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91: stopping tablet replica
I20260925 13:21:09.447628 17895 tablet_replica.cc:330] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea: stopping tablet replica
I20260925 13:21:09.447719 17895 raft_consensus.cc:2229] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:21:09.447821 17895 raft_consensus.cc:2258] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:09.447813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 17455
I20260925 13:21:09.449249 17894 raft_consensus.cc:2229] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:21:09.449373 17894 raft_consensus.cc:2258] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:09.449798 17894 ts_tablet_manager.cc:1905] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:21:09.451288 17894 ts_tablet_manager.cc:1918] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:21:09.451344 17894 log.cc:1196] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/cd24b97aa5374292858ac71fa6bc7aed
I20260925 13:21:09.451563 17894 ts_tablet_manager.cc:1939] T cd24b97aa5374292858ac71fa6bc7aed P f122179d4bb342aa94a59eea35597b91: Deleting consensus metadata
I20260925 13:21:09.451936 17377 catalog_manager.cc:4841] TS f122179d4bb342aa94a59eea35597b91 (127.0.98.3:45409): tablet cd24b97aa5374292858ac71fa6bc7aed (table test-table [id=81bc0bf2d338471d9f58a5323a1fe04a]) successfully deleted
W20260925 13:21:09.452924 17379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38687 (error 108)
I20260925 13:21:09.453125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 17589
I20260925 13:21:09.453339 17895 ts_tablet_manager.cc:1905] T cd24b97aa5374292858ac71fa6bc7aed P ca69a30e07224fc39e31d1a89a4d16ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:21:09.453871 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.453710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38687 (local address 127.0.0.1:56234)
0925 13:21:09.453778 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:09.453834 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 13:21:09.453976 17379 catalog_manager.cc:4584] TS 633b9cffbaf046299be77b1df5bcdab8 (127.0.98.1:38687): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cd24b97aa5374292858ac71fa6bc7aed: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38687: connect: Connection refused (error 111)
I20260925 13:21:09.458357 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 17723
I20260925 13:21:09.458400 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.458294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46877 (local address 127.0.0.1:34412)
0925 13:21:09.458344 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:09.458367 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 13:21:09.464485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 17361
2026-09-25T13:21:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2067 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T13:21:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:09Z Disabled control of system clock
I20260925 13:21:09.490437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36615
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:21:09.569715 17900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:09.569964 17900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:09.569985 17900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:09.571264 17900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:09.571318 17900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:09.571329 17900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:09.571341 17900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:09.571352 17900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:09.572781 17900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:09.573045 17900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:09.573349 17900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:09.577900 17905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:09.577907 17908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:09.577884 17906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:09.578227 17900 server_base.cc:1034] running on GCE node
I20260925 13:21:09.578487 17900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:09.578764 17900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:09.579918 17900 hybrid_clock.cc:648] HybridClock initialized: now 1790342469579901 us; error 35 us; skew 500 ppm
I20260925 13:21:09.581185 17900 webserver.cc:460] Webserver started at http://127.0.98.62:38975/ using document root <none> and password file <none>
I20260925 13:21:09.581416 17900 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:09.581461 17900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:09.581617 17900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:09.582692 17900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "931db9087a3a4463b857fcee23793d92"
format_stamp: "Formatted at 2026-09-25 13:21:09 on dist-test-slave-t8d9"
server_key: "d8c014739204427d0f23625b6c16fb10"
server_key_iv: "be0d7be65d1b18d62b4d5c8fc77366c6"
server_key_version: "encryptionkey@0"
I20260925 13:21:09.583019 17900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "931db9087a3a4463b857fcee23793d92"
format_stamp: "Formatted at 2026-09-25 13:21:09 on dist-test-slave-t8d9"
server_key: "d8c014739204427d0f23625b6c16fb10"
server_key_iv: "be0d7be65d1b18d62b4d5c8fc77366c6"
server_key_version: "encryptionkey@0"
I20260925 13:21:09.584280 17900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 13:21:09.585029 17914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:09.585179 17900 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:21:09.585259 17900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "931db9087a3a4463b857fcee23793d92"
format_stamp: "Formatted at 2026-09-25 13:21:09 on dist-test-slave-t8d9"
server_key: "d8c014739204427d0f23625b6c16fb10"
server_key_iv: "be0d7be65d1b18d62b4d5c8fc77366c6"
server_key_version: "encryptionkey@0"
I20260925 13:21:09.585330 17900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:09.608171 17900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:09.608500 17900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:09.608629 17900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:09.612468 17900 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36615
I20260925 13:21:09.612510 17966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36615 every 8 connection(s)
I20260925 13:21:09.612846 17900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:09.613852 17967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:09.614565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 17900
I20260925 13:21:09.614660 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:09.614836 392 external_mini_cluster.cc:1428] Setting key f2ea3e59b82e685725094871463cd13a
I20260925 13:21:09.616982 17967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92: Bootstrap starting.
I20260925 13:21:09.617725 17967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:09.618036 17967 log.cc:824] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:09.618741 17967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92: No bootstrap required, opened a new log
I20260925 13:21:09.620936 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.615379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.0.98.62:36615)
0925 13:21:09.615593 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:09.615603 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:09.615636 (+ 33us) server_negotiation.cc:407] Connection header received
0925 13:21:09.616128 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:09.616138 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:09.616227 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:09.616380 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:09.616564 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.617898 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.618842 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.619441 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.619577 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.620163 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:09.620168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:09.620169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:09.620371 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:09.620418 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:09.620433 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:09.620507 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:09.620599 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 13:21:09.620689 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260925 13:21:09.621042 17967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931db9087a3a4463b857fcee23793d92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36615 } }
I20260925 13:21:09.621184 17967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:09.621215 17967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931db9087a3a4463b857fcee23793d92, State: Initialized, Role: FOLLOWER
I20260925 13:21:09.621575 17967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931db9087a3a4463b857fcee23793d92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36615 } }
I20260925 13:21:09.621663 17967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:09.621703 17967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:09.621749 17967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:09.622275 17967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931db9087a3a4463b857fcee23793d92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36615 } }
I20260925 13:21:09.622381 17967 leader_election.cc:302] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 931db9087a3a4463b857fcee23793d92; no voters:
I20260925 13:21:09.622547 17967 leader_election.cc:288] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:09.622589 17972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:09.622735 17972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [term 1 LEADER]: Becoming Leader. State: Replica: 931db9087a3a4463b857fcee23793d92, State: Running, Role: LEADER
I20260925 13:21:09.622807 17967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:09.622829 17972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931db9087a3a4463b857fcee23793d92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36615 } }
I20260925 13:21:09.623179 17973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "931db9087a3a4463b857fcee23793d92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931db9087a3a4463b857fcee23793d92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36615 } } }
I20260925 13:21:09.623193 17974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 931db9087a3a4463b857fcee23793d92. Latest consensus state: current_term: 1 leader_uuid: "931db9087a3a4463b857fcee23793d92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931db9087a3a4463b857fcee23793d92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36615 } } }
I20260925 13:21:09.623377 17974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:09.623560 17981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:09.623562 17973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:09.624099 17981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:09.625499 17981 catalog_manager.cc:1347] Generated new cluster ID: e34bb1e16840430992d831a1d88743e9
I20260925 13:21:09.625581 17981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:09.629176 17981 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:09.629776 17981 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:09.641258 17981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92: Generated new TSK 0
I20260925 13:21:09.641463 17981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:09.649685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36615
--builtin_ntp_servers=127.0.98.20:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:21:09.767819 17991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:09.768139 17991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:09.768175 17991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:09.770010 17991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:09.770077 17991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:09.770119 17991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:21:09.772181 17991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:09.772471 17991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:09.772861 17991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:09.777994 17996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:09.778084 17997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:09.778182 17999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:09.778395 17991 server_base.cc:1034] running on GCE node
I20260925 13:21:09.778587 17991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:09.778803 17991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:09.779937 17991 hybrid_clock.cc:648] HybridClock initialized: now 1790342469779921 us; error 30 us; skew 500 ppm
I20260925 13:21:09.781155 17991 webserver.cc:460] Webserver started at http://127.0.98.1:35403/ using document root <none> and password file <none>
I20260925 13:21:09.781378 17991 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:09.781442 17991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:09.781616 17991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:09.782661 17991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "b51ef900e15f42569fa357183342c56b"
format_stamp: "Formatted at 2026-09-25 13:21:09 on dist-test-slave-t8d9"
server_key: "6ddd4bf5ad0ee088ae2088ea8e85d448"
server_key_iv: "46eda23be62eb27ba70289dc198c2918"
server_key_version: "encryptionkey@0"
I20260925 13:21:09.782994 17991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b51ef900e15f42569fa357183342c56b"
format_stamp: "Formatted at 2026-09-25 13:21:09 on dist-test-slave-t8d9"
server_key: "6ddd4bf5ad0ee088ae2088ea8e85d448"
server_key_iv: "46eda23be62eb27ba70289dc198c2918"
server_key_version: "encryptionkey@0"
I20260925 13:21:09.784258 17991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:21:09.784963 18005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:09.785123 17991 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:09.785210 17991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "b51ef900e15f42569fa357183342c56b"
format_stamp: "Formatted at 2026-09-25 13:21:09 on dist-test-slave-t8d9"
server_key: "6ddd4bf5ad0ee088ae2088ea8e85d448"
server_key_iv: "46eda23be62eb27ba70289dc198c2918"
server_key_version: "encryptionkey@0"
I20260925 13:21:09.785271 17991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:09.802393 17991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:09.802697 17991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:09.802824 17991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:09.803033 17991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:09.803356 17991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:09.803390 17991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:09.803418 17991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:09.803444 17991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:09.808854 17991 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35443
I20260925 13:21:09.808900 18118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35443 every 8 connection(s)
I20260925 13:21:09.809212 17991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:21:09.814095 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.809893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58015 (local address 127.0.98.62:36615)
0925 13:21:09.809936 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:09.809938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:09.810162 (+ 224us) server_negotiation.cc:407] Connection header received
0925 13:21:09.810633 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:09.810634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:09.810648 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:09.810694 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:09.811320 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.811812 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.812719 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.812890 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.812933 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.813858 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:09.813859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:09.813860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:09.813864 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:09.813879 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:09.813883 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:09.813935 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:09.814001 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 13:21:09.814031 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:21:09.814281 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.809881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36615 (local address 127.0.98.1:58015)
0925 13:21:09.810046 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:09.810080 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:09.810558 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:09.810752 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:09.810755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:09.810764 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:09.811261 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.811266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.811897 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.811897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.812659 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.812661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.812984 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.812985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.813111 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.813709 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:21:09.813709 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:09.813740 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:21:09.813763 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:09.813943 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:09.813946 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:09.813949 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:09.813996 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 13:21:09.814083 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260925 13:21:09.814791 18119 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36615
I20260925 13:21:09.814881 18119 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:09.815061 18119 heartbeater.cc:511] Master 127.0.98.62:36615 requested a full tablet report, sending...
I20260925 13:21:09.815237 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 17991
I20260925 13:21:09.815333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:21:09.815536 392 external_mini_cluster.cc:1428] Setting key 47f761df8724caa2840aa2c0a4affe62
I20260925 13:21:09.815584 17931 ts_manager.cc:194] Registered new tserver with Master: b51ef900e15f42569fa357183342c56b (127.0.98.1:35443)
I20260925 13:21:09.816418 17931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58015
I20260925 13:21:09.816480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36615
--builtin_ntp_servers=127.0.98.20:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:21:09.896154 18122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:09.896380 18122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:09.896409 18122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:09.897616 18122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:09.897655 18122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:09.897687 18122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:21:09.899055 18122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:09.899287 18122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:09.899570 18122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:09.904386 18128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:09.904388 18130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:09.904381 18127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:09.904558 18122 server_base.cc:1034] running on GCE node
I20260925 13:21:09.904878 18122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:09.905145 18122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:09.906303 18122 hybrid_clock.cc:648] HybridClock initialized: now 1790342469906283 us; error 38 us; skew 500 ppm
I20260925 13:21:09.907564 18122 webserver.cc:460] Webserver started at http://127.0.98.2:37655/ using document root <none> and password file <none>
I20260925 13:21:09.907784 18122 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:09.907835 18122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:09.907981 18122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:09.909075 18122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "b29620702ccc461bbe9c07ab72afa407"
format_stamp: "Formatted at 2026-09-25 13:21:09 on dist-test-slave-t8d9"
server_key: "845b1a5624bbfd2e79aeeed12c715250"
server_key_iv: "1aa5957dc80110cf3343c35dfe7e96fc"
server_key_version: "encryptionkey@0"
I20260925 13:21:09.909392 18122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b29620702ccc461bbe9c07ab72afa407"
format_stamp: "Formatted at 2026-09-25 13:21:09 on dist-test-slave-t8d9"
server_key: "845b1a5624bbfd2e79aeeed12c715250"
server_key_iv: "1aa5957dc80110cf3343c35dfe7e96fc"
server_key_version: "encryptionkey@0"
I20260925 13:21:09.910689 18122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:21:09.911453 18136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:09.911607 18122 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:21:09.911686 18122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "b29620702ccc461bbe9c07ab72afa407"
format_stamp: "Formatted at 2026-09-25 13:21:09 on dist-test-slave-t8d9"
server_key: "845b1a5624bbfd2e79aeeed12c715250"
server_key_iv: "1aa5957dc80110cf3343c35dfe7e96fc"
server_key_version: "encryptionkey@0"
I20260925 13:21:09.911747 18122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:09.926829 18122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:09.927150 18122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:09.927280 18122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:09.927498 18122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:09.927814 18122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:09.927845 18122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:09.927873 18122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:09.927906 18122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:09.933228 18122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42609
I20260925 13:21:09.933288 18249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42609 every 8 connection(s)
I20260925 13:21:09.933642 18122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:21:09.938520 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.934295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41545 (local address 127.0.98.62:36615)
0925 13:21:09.934369 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:09.934372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:09.934586 (+ 214us) server_negotiation.cc:407] Connection header received
0925 13:21:09.935118 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:09.935119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:09.935130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:09.935179 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:09.935817 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.936306 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.937255 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.937435 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.937479 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.938313 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:09.938314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:09.938314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:09.938318 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:09.938329 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:09.938333 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:09.938370 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:09.938439 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 13:21:09.938459 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 13:21:09.938732 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:09.934289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36615 (local address 127.0.98.2:41545)
0925 13:21:09.934471 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:09.934506 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:09.935033 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:09.935239 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:09.935242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:09.935251 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:09.935754 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.935759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.936370 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.936370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.937190 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:09.937192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:09.937501 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:09.937502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:09.937670 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:09.938190 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:21:09.938191 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:09.938221 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:21:09.938246 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:09.938379 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:09.938382 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:09.938384 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:21:09.938436 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 13:21:09.938519 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260925 13:21:09.939195 18250 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36615
I20260925 13:21:09.939284 18250 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:09.939464 18250 heartbeater.cc:511] Master 127.0.98.62:36615 requested a full tablet report, sending...
I20260925 13:21:09.939798 17931 ts_manager.cc:194] Registered new tserver with Master: b29620702ccc461bbe9c07ab72afa407 (127.0.98.2:42609)
I20260925 13:21:09.940160 17931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41545
I20260925 13:21:09.941392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 18122
I20260925 13:21:09.941486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:21:09.941680 392 external_mini_cluster.cc:1428] Setting key ae71307c0e91d7045384c4fb065b787a
I20260925 13:21:09.942519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36615
--builtin_ntp_servers=127.0.98.20:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:21:10.023411 18253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:10.023675 18253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:10.023702 18253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:10.025028 18253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:10.025079 18253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:10.025116 18253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:21:10.026739 18253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:10.027051 18253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:10.027354 18253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:10.032212 18261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:10.032212 18259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:10.032222 18258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:10.032481 18253 server_base.cc:1034] running on GCE node
I20260925 13:21:10.032732 18253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:10.033013 18253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:10.034179 18253 hybrid_clock.cc:648] HybridClock initialized: now 1790342470034142 us; error 56 us; skew 500 ppm
I20260925 13:21:10.035552 18253 webserver.cc:460] Webserver started at http://127.0.98.3:41605/ using document root <none> and password file <none>
I20260925 13:21:10.035781 18253 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:10.035840 18253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:10.035966 18253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:10.037065 18253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "906fa837811343c68a690116f3bf70a1"
format_stamp: "Formatted at 2026-09-25 13:21:10 on dist-test-slave-t8d9"
server_key: "88a9bee75539a352a481b987dd3a4fb8"
server_key_iv: "2a13568e5912b34d3dd9a5d5d79462fd"
server_key_version: "encryptionkey@0"
I20260925 13:21:10.037415 18253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "906fa837811343c68a690116f3bf70a1"
format_stamp: "Formatted at 2026-09-25 13:21:10 on dist-test-slave-t8d9"
server_key: "88a9bee75539a352a481b987dd3a4fb8"
server_key_iv: "2a13568e5912b34d3dd9a5d5d79462fd"
server_key_version: "encryptionkey@0"
I20260925 13:21:10.038808 18253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:21:10.039613 18267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:10.039851 18253 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:21:10.039937 18253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "906fa837811343c68a690116f3bf70a1"
format_stamp: "Formatted at 2026-09-25 13:21:10 on dist-test-slave-t8d9"
server_key: "88a9bee75539a352a481b987dd3a4fb8"
server_key_iv: "2a13568e5912b34d3dd9a5d5d79462fd"
server_key_version: "encryptionkey@0"
I20260925 13:21:10.040004 18253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:10.052042 18253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:10.052366 18253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:10.052500 18253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:10.052739 18253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:10.053064 18253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:10.053098 18253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:10.053129 18253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:10.053153 18253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:10.058799 18253 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36845
I20260925 13:21:10.058840 18380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36845 every 8 connection(s)
I20260925 13:21:10.059171 18253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:21:10.064321 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:10.059859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56663 (local address 127.0.98.62:36615)
0925 13:21:10.059918 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:10.059920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:10.060163 (+ 243us) server_negotiation.cc:407] Connection header received
0925 13:21:10.060672 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:10.060674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:10.060686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:10.060730 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:10.061409 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.061936 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.062924 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.063118 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.063173 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:10.064035 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:10.064037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:10.064037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:10.064043 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:10.064058 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:10.064062 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:10.064104 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:10.064189 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 13:21:10.064221 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 13:21:10.064515 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:10.059852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36615 (local address 127.0.98.3:56663)
0925 13:21:10.060040 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:10.060073 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:10.060568 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:10.060810 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:10.060813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:10.060822 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:10.061318 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:10.061323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.062031 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.062032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:10.062827 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:10.062828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.063191 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.063192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:10.063320 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:10.063871 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:21:10.063872 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:10.063902 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:21:10.063927 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:10.064112 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:10.064117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:10.064121 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:21:10.064182 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 13:21:10.064290 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260925 13:21:10.065125 18381 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36615
I20260925 13:21:10.065223 18381 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:10.065404 18381 heartbeater.cc:511] Master 127.0.98.62:36615 requested a full tablet report, sending...
I20260925 13:21:10.065869 17931 ts_manager.cc:194] Registered new tserver with Master: 906fa837811343c68a690116f3bf70a1 (127.0.98.3:36845)
I20260925 13:21:10.066219 17931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56663
I20260925 13:21:10.067101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 18253
I20260925 13:21:10.067193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:21:10.067353 392 external_mini_cluster.cc:1428] Setting key a28394cd7f1389788eab93adf7106592
I20260925 13:21:10.068153 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:21:10.072126 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:10.069440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.0.98.62:36615)
0925 13:21:10.069506 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:10.069508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:10.069637 (+ 129us) server_negotiation.cc:407] Connection header received
0925 13:21:10.069674 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:10.069675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:10.069683 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:10.069730 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:10.069859 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.070339 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.071041 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.071237 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.071318 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:10.071921 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:10.071922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:10.071923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:10.071926 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:10.071941 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:10.071944 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:10.071985 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:10.072043 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 13:21:10.072064 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 13:21:10.073487 17931 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:21:10.075076 17931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:21:10.081264 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:10.077236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35443 (local address 127.0.0.1:41112)
0925 13:21:10.077666 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:10.077676 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:10.077743 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:10.078067 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:10.078068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:10.078071 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:10.078140 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:10.078142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.079044 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.079045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:10.080188 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:10.080190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.080969 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.080970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:10.081044 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:10.081057 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 13:21:10.081127 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 13:21:10.081180 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"spinlock_wait_cycles":680320,"thread_start_us":373,"threads_started":1}
I20260925 13:21:10.081274 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:10.077402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.0.98.1:35443)
0925 13:21:10.077592 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:10.077597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:10.077730 (+ 133us) server_negotiation.cc:407] Connection header received
0925 13:21:10.077812 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:10.077814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:10.077826 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:10.077893 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:10.078215 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.078964 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.080272 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.080896 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.080958 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:10.081056 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:10.081143 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 13:21:10.081181 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260925 13:21:10.081743 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:10.077493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.0.98.3:36845)
0925 13:21:10.077738 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:10.077743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:10.077893 (+ 150us) server_negotiation.cc:407] Connection header received
0925 13:21:10.078045 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:10.078048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:10.078064 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:10.078134 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:10.078297 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.079399 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.080593 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.081281 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.081333 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:10.081411 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:10.081615 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 13:21:10.081661 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":115,"threads_started":1}
I20260925 13:21:10.081735 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:10.077237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36845 (local address 127.0.0.1:44820)
0925 13:21:10.077832 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:10.077840 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:10.077918 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:10.078198 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:10.078199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:10.078201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:10.078237 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:10.078239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.079483 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.079484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:10.080460 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:10.080463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.081448 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.081449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:10.081522 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:10.081525 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:10.081595 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 13:21:10.081642 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":38,"spinlock_wait_cycles":896,"thread_start_us":125,"threads_started":1}
I20260925 13:21:10.081735 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:10.077236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42609 (local address 127.0.0.1:39228)
0925 13:21:10.077422 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:10.077467 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:10.077578 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:10.078057 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:10.078060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:10.078061 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:10.078138 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:10.078141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.079475 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.079476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:10.080455 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:10.080457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.081449 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.081450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:10.081515 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:10.081518 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:10.081595 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 13:21:10.081642 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":6,"thread_start_us":74,"threads_started":1}
I20260925 13:21:10.082007 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:10.077336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.0.98.2:42609)
0925 13:21:10.077828 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:10.077831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:10.077842 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:21:10.077884 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:10.077886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:10.077902 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:10.077978 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:10.078216 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.079367 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.080525 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.081252 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.081323 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:10.081411 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:10.081882 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 13:21:10.081928 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":85,"threads_started":1}
I20260925 13:21:10.082480 18315 tablet_service.cc:1467] Processing CreateTablet for tablet 90e2a6bd43f74ee584a0f9826c69f4ea (DEFAULT_TABLE table=test-table [id=8eeb33106a32490cba99cb0c557a3d88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:10.082458 18053 tablet_service.cc:1467] Processing CreateTablet for tablet 90e2a6bd43f74ee584a0f9826c69f4ea (DEFAULT_TABLE table=test-table [id=8eeb33106a32490cba99cb0c557a3d88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:10.082824 18315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e2a6bd43f74ee584a0f9826c69f4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:10.082824 18053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e2a6bd43f74ee584a0f9826c69f4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:10.086227 18396 tablet_bootstrap.cc:492] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b: Bootstrap starting.
I20260925 13:21:10.086469 18395 tablet_bootstrap.cc:492] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1: Bootstrap starting.
I20260925 13:21:10.086799 18184 tablet_service.cc:1467] Processing CreateTablet for tablet 90e2a6bd43f74ee584a0f9826c69f4ea (DEFAULT_TABLE table=test-table [id=8eeb33106a32490cba99cb0c557a3d88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:10.087098 18184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e2a6bd43f74ee584a0f9826c69f4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:10.087103 18395 tablet_bootstrap.cc:654] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:10.087358 18395 log.cc:824] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:10.088147 18396 tablet_bootstrap.cc:654] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:10.088184 18395 tablet_bootstrap.cc:492] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1: No bootstrap required, opened a new log
I20260925 13:21:10.088243 18395 ts_tablet_manager.cc:1397] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:21:10.088485 18396 log.cc:824] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:10.089269 18396 tablet_bootstrap.cc:492] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b: No bootstrap required, opened a new log
I20260925 13:21:10.089339 18396 ts_tablet_manager.cc:1397] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 13:21:10.090204 18399 tablet_bootstrap.cc:492] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407: Bootstrap starting.
I20260925 13:21:10.090821 18399 tablet_bootstrap.cc:654] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:10.090827 18395 raft_consensus.cc:348] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b29620702ccc461bbe9c07ab72afa407" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42609 } } peers { permanent_uuid: "906fa837811343c68a690116f3bf70a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36845 } } peers { permanent_uuid: "b51ef900e15f42569fa357183342c56b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35443 } }
I20260925 13:21:10.091006 18395 raft_consensus.cc:374] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:10.091049 18395 raft_consensus.cc:729] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906fa837811343c68a690116f3bf70a1, State: Initialized, Role: FOLLOWER
I20260925 13:21:10.091109 18399 log.cc:824] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:10.091164 18395 consensus_queue.cc:260] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b29620702ccc461bbe9c07ab72afa407" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42609 } } peers { permanent_uuid: "906fa837811343c68a690116f3bf70a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36845 } } peers { permanent_uuid: "b51ef900e15f42569fa357183342c56b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35443 } }
I20260925 13:21:10.091435 18395 ts_tablet_manager.cc:1428] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 13:21:10.091467 18381 heartbeater.cc:503] Master 127.0.98.62:36615 was elected leader, sending a full tablet report...
I20260925 13:21:10.091755 18399 tablet_bootstrap.cc:492] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407: No bootstrap required, opened a new log
I20260925 13:21:10.091817 18399 ts_tablet_manager.cc:1397] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:21:10.091969 18396 raft_consensus.cc:348] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b29620702ccc461bbe9c07ab72afa407" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42609 } } peers { permanent_uuid: "906fa837811343c68a690116f3bf70a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36845 } } peers { permanent_uuid: "b51ef900e15f42569fa357183342c56b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35443 } }
I20260925 13:21:10.092087 18396 raft_consensus.cc:374] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:10.092104 18396 raft_consensus.cc:729] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51ef900e15f42569fa357183342c56b, State: Initialized, Role: FOLLOWER
I20260925 13:21:10.092190 18396 consensus_queue.cc:260] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b29620702ccc461bbe9c07ab72afa407" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42609 } } peers { permanent_uuid: "906fa837811343c68a690116f3bf70a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36845 } } peers { permanent_uuid: "b51ef900e15f42569fa357183342c56b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35443 } }
I20260925 13:21:10.092567 18396 ts_tablet_manager.cc:1428] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:21:10.092607 18119 heartbeater.cc:503] Master 127.0.98.62:36615 was elected leader, sending a full tablet report...
I20260925 13:21:10.094076 18399 raft_consensus.cc:348] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b29620702ccc461bbe9c07ab72afa407" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42609 } } peers { permanent_uuid: "906fa837811343c68a690116f3bf70a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36845 } } peers { permanent_uuid: "b51ef900e15f42569fa357183342c56b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35443 } }
I20260925 13:21:10.094213 18399 raft_consensus.cc:374] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:10.094233 18399 raft_consensus.cc:729] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b29620702ccc461bbe9c07ab72afa407, State: Initialized, Role: FOLLOWER
I20260925 13:21:10.094316 18399 consensus_queue.cc:260] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b29620702ccc461bbe9c07ab72afa407" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42609 } } peers { permanent_uuid: "906fa837811343c68a690116f3bf70a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36845 } } peers { permanent_uuid: "b51ef900e15f42569fa357183342c56b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35443 } }
I20260925 13:21:10.094525 18399 ts_tablet_manager.cc:1428] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:21:10.094578 18250 heartbeater.cc:503] Master 127.0.98.62:36615 was elected leader, sending a full tablet report...
W20260925 13:21:10.184199 18251 tablet.cc:2369] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:21:10.237936 18402 raft_consensus.cc:482] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:21:10.238081 18402 raft_consensus.cc:504] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b29620702ccc461bbe9c07ab72afa407" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42609 } } peers { permanent_uuid: "906fa837811343c68a690116f3bf70a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36845 } } peers { permanent_uuid: "b51ef900e15f42569fa357183342c56b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35443 } }
I20260925 13:21:10.238456 18402 leader_election.cc:288] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b29620702ccc461bbe9c07ab72afa407 (127.0.98.2:42609), 906fa837811343c68a690116f3bf70a1 (127.0.98.3:36845)
I20260925 13:21:10.241078 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:10.238754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54645 (local address 127.0.98.3:36845)
0925 13:21:10.238810 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:10.238812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:10.238821 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:21:10.238850 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:10.238851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:10.238859 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:10.238909 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:10.239050 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.239526 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.240409 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.240764 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.240823 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:10.240834 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:10.240982 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 13:21:10.241017 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:21:10.241077 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:10.238645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36845 (local address 127.0.98.1:54645)
0925 13:21:10.238699 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:10.238712 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:10.238777 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:10.238918 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:10.238919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:10.238920 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:10.238987 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:10.238989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.239596 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.239596 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:10.240320 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:10.240322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.240830 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.240831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:10.240903 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:10.240906 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:10.240960 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 13:21:10.241001 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 13:21:10.241190 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:10.238645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42609 (local address 127.0.98.1:59373)
0925 13:21:10.238826 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:10.238836 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:10.238895 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:10.239028 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:10.239029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:10.239030 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:10.239096 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:10.239098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.239677 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.239678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:10.240419 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:10.240421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.240939 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.240940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:10.241004 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:10.241007 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:10.241050 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:21:10.241089 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":15,"thread_start_us":106,"threads_started":1}
I20260925 13:21:10.241176 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:10.238756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59373 (local address 127.0.98.2:42609)
0925 13:21:10.238831 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:10.238833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:10.238934 (+ 101us) server_negotiation.cc:407] Connection header received
0925 13:21:10.238964 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:10.238965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:10.238974 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:10.239017 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:10.239155 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.239607 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.240509 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.240861 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.240923 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:10.240933 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:10.241066 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 13:21:10.241103 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 13:21:10.241415 18335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e2a6bd43f74ee584a0f9826c69f4ea" candidate_uuid: "b51ef900e15f42569fa357183342c56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906fa837811343c68a690116f3bf70a1" is_pre_election: true
I20260925 13:21:10.241483 18204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e2a6bd43f74ee584a0f9826c69f4ea" candidate_uuid: "b51ef900e15f42569fa357183342c56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b29620702ccc461bbe9c07ab72afa407" is_pre_election: true
I20260925 13:21:10.241600 18335 raft_consensus.cc:2454] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b51ef900e15f42569fa357183342c56b in term 0.
I20260925 13:21:10.241649 18204 raft_consensus.cc:2454] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b51ef900e15f42569fa357183342c56b in term 0.
I20260925 13:21:10.241820 18006 leader_election.cc:302] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 906fa837811343c68a690116f3bf70a1, b51ef900e15f42569fa357183342c56b; no voters:
I20260925 13:21:10.241951 18402 raft_consensus.cc:2781] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:21:10.242009 18402 raft_consensus.cc:482] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:21:10.242031 18402 raft_consensus.cc:3037] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:10.242726 18402 raft_consensus.cc:504] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b29620702ccc461bbe9c07ab72afa407" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42609 } } peers { permanent_uuid: "906fa837811343c68a690116f3bf70a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36845 } } peers { permanent_uuid: "b51ef900e15f42569fa357183342c56b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35443 } }
I20260925 13:21:10.242889 18402 leader_election.cc:288] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [CANDIDATE]: Term 1 election: Requested vote from peers b29620702ccc461bbe9c07ab72afa407 (127.0.98.2:42609), 906fa837811343c68a690116f3bf70a1 (127.0.98.3:36845)
I20260925 13:21:10.243040 18335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e2a6bd43f74ee584a0f9826c69f4ea" candidate_uuid: "b51ef900e15f42569fa357183342c56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906fa837811343c68a690116f3bf70a1"
I20260925 13:21:10.243042 18204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e2a6bd43f74ee584a0f9826c69f4ea" candidate_uuid: "b51ef900e15f42569fa357183342c56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b29620702ccc461bbe9c07ab72afa407"
I20260925 13:21:10.243110 18204 raft_consensus.cc:3037] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:10.243115 18335 raft_consensus.cc:3037] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:10.243779 18204 raft_consensus.cc:2454] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b51ef900e15f42569fa357183342c56b in term 1.
I20260925 13:21:10.243782 18335 raft_consensus.cc:2454] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b51ef900e15f42569fa357183342c56b in term 1.
I20260925 13:21:10.243920 18006 leader_election.cc:302] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 906fa837811343c68a690116f3bf70a1, b51ef900e15f42569fa357183342c56b; no voters:
I20260925 13:21:10.244006 18402 raft_consensus.cc:2781] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:10.244153 18402 raft_consensus.cc:686] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [term 1 LEADER]: Becoming Leader. State: Replica: b51ef900e15f42569fa357183342c56b, State: Running, Role: LEADER
I20260925 13:21:10.244236 18402 consensus_queue.cc:237] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b29620702ccc461bbe9c07ab72afa407" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42609 } } peers { permanent_uuid: "906fa837811343c68a690116f3bf70a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36845 } } peers { permanent_uuid: "b51ef900e15f42569fa357183342c56b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35443 } }
I20260925 13:21:10.245039 17931 catalog_manager.cc:5495] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b reported cstate change: term changed from 0 to 1, leader changed from <none> to b51ef900e15f42569fa357183342c56b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b51ef900e15f42569fa357183342c56b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b29620702ccc461bbe9c07ab72afa407" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "906fa837811343c68a690116f3bf70a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b51ef900e15f42569fa357183342c56b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35443 } health_report { overall_health: HEALTHY } } }
I20260925 13:21:10.263826 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:10.261521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.0.98.1:35443)
0925 13:21:10.261599 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:10.261601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:10.261608 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:21:10.261650 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:10.261651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:10.261672 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:10.261732 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:10.261880 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.262361 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.263084 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:10.263300 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:10.263362 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:10.263493 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:10.263607 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:10.263678 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:10.263697 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 13:21:10.263719 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 13:21:10.265772 18335 raft_consensus.cc:1261] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1 [term 1 FOLLOWER]: Refusing update from remote peer b51ef900e15f42569fa357183342c56b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:10.265772 18204 raft_consensus.cc:1261] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407 [term 1 FOLLOWER]: Refusing update from remote peer b51ef900e15f42569fa357183342c56b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:10.266213 18405 consensus_queue.cc:1035] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29620702ccc461bbe9c07ab72afa407" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:21:10.266364 18402 consensus_queue.cc:1035] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [LEADER]: Connected to new peer: Peer: permanent_uuid: "906fa837811343c68a690116f3bf70a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:21:10.269169 18410 mvcc.cc:205] Tried to move back new op lower bound from 7333242758205665280 to 7333242758121078784. Current Snapshot: MvccSnapshot[applied={T|T < 7333242758205665280}]
I20260925 13:21:10.269292 18409 mvcc.cc:205] Tried to move back new op lower bound from 7333242758205665280 to 7333242758121078784. Current Snapshot: MvccSnapshot[applied={T|T < 7333242758205665280}]
I20260925 13:21:10.271190 17931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:21:10.271297 17931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:21:10.272046 17931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 931db9087a3a4463b857fcee23793d92: Sending DeleteTablet for 3 replicas of tablet 90e2a6bd43f74ee584a0f9826c69f4ea
I20260925 13:21:10.272339 18184 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e2a6bd43f74ee584a0f9826c69f4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:10 UTC) from {username='slave'} at 127.0.0.1:39228
I20260925 13:21:10.272368 18315 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e2a6bd43f74ee584a0f9826c69f4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:10 UTC) from {username='slave'} at 127.0.0.1:44820
I20260925 13:21:10.272598 18418 tablet_replica.cc:330] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1: stopping tablet replica
I20260925 13:21:10.272698 18418 raft_consensus.cc:2229] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:21:10.272703 18053 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e2a6bd43f74ee584a0f9826c69f4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:10 UTC) from {username='slave'} at 127.0.0.1:41112
I20260925 13:21:10.272797 18418 raft_consensus.cc:2258] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:10.272903 18419 tablet_replica.cc:330] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b: stopping tablet replica
I20260925 13:21:10.272989 18419 raft_consensus.cc:2229] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:21:10.273025 18417 tablet_replica.cc:330] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407: stopping tablet replica
I20260925 13:21:10.273054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 17991
I20260925 13:21:10.273078 18419 raft_consensus.cc:2258] T 90e2a6bd43f74ee584a0f9826c69f4ea P b51ef900e15f42569fa357183342c56b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:10.273118 18417 raft_consensus.cc:2229] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:21:10.273551 18417 raft_consensus.cc:2258] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:10.274762 18417 ts_tablet_manager.cc:1905] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:21:10.274763 18418 ts_tablet_manager.cc:1905] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:21:10.276407 18417 ts_tablet_manager.cc:1918] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:21:10.276403 18418 ts_tablet_manager.cc:1918] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:21:10.276477 18417 log.cc:1196] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/90e2a6bd43f74ee584a0f9826c69f4ea
I20260925 13:21:10.276481 18418 log.cc:1196] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/90e2a6bd43f74ee584a0f9826c69f4ea
I20260925 13:21:10.276710 18417 ts_tablet_manager.cc:1939] T 90e2a6bd43f74ee584a0f9826c69f4ea P b29620702ccc461bbe9c07ab72afa407: Deleting consensus metadata
I20260925 13:21:10.276715 18418 ts_tablet_manager.cc:1939] T 90e2a6bd43f74ee584a0f9826c69f4ea P 906fa837811343c68a690116f3bf70a1: Deleting consensus metadata
I20260925 13:21:10.277115 17915 catalog_manager.cc:4841] TS 906fa837811343c68a690116f3bf70a1 (127.0.98.3:36845): tablet 90e2a6bd43f74ee584a0f9826c69f4ea (table test-table [id=8eeb33106a32490cba99cb0c557a3d88]) successfully deleted
I20260925 13:21:10.277228 17916 catalog_manager.cc:4841] TS b29620702ccc461bbe9c07ab72afa407 (127.0.98.2:42609): tablet 90e2a6bd43f74ee584a0f9826c69f4ea (table test-table [id=8eeb33106a32490cba99cb0c557a3d88]) successfully deleted
W20260925 13:21:10.278679 17917 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35443 (error 108)
I20260925 13:21:10.278873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 18122
I20260925 13:21:10.282348 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:10.282184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35443 (local address 127.0.0.1:41138)
0925 13:21:10.282251 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:10.282309 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 13:21:10.282825 17917 catalog_manager.cc:4584] TS b51ef900e15f42569fa357183342c56b (127.0.98.1:35443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90e2a6bd43f74ee584a0f9826c69f4ea: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35443: connect: Connection refused (error 111)
I20260925 13:21:10.283165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 18253
I20260925 13:21:10.287171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 17900
2026-09-25T13:21:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (821 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:10 dist-test-slave-t8d9 krb5kdc[18423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:10 dist-test-slave-t8d9 krb5kdc[18423](info): set up 2 sockets
Sep 25 13:21:10 dist-test-slave-t8d9 krb5kdc[18423](info): commencing operation
krb5kdc: starting...
W20260925 13:21:11.237160 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:11 dist-test-slave-t8d9 krb5kdc[18423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:21:11.366148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38149
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:11.446437 18436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:11.446640 18436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:11.446659 18436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:11.447762 18436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:11.447795 18436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:11.447808 18436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:11.447819 18436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:11.447832 18436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:11.449246 18436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:11.449442 18436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:11.449645 18436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:11.454216 18444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:11.454214 18442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:11.454434 18441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:11.454402 18436 server_base.cc:1034] running on GCE node
I20260925 13:21:11.454665 18436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:11.454941 18436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:11.456099 18436 hybrid_clock.cc:648] HybridClock initialized: now 1790342471456082 us; error 30 us; skew 500 ppm
Sep 25 13:21:11 dist-test-slave-t8d9 krb5kdc[18423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:11.458603 18436 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:21:11.459072 18436 webserver.cc:460] Webserver started at http://127.0.98.62:42327/ using document root <none> and password file <none>
I20260925 13:21:11.459280 18436 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:11.459328 18436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:11.459451 18436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:11.460587 18436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "8a16eae443b347e0ac8b101efb359a64"
format_stamp: "Formatted at 2026-09-25 13:21:11 on dist-test-slave-t8d9"
server_key: "6783c27a828c46830018672407e94808"
server_key_iv: "6581c6202806cc693a010fbc62c3e74e"
server_key_version: "encryptionkey@0"
I20260925 13:21:11.460935 18436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "8a16eae443b347e0ac8b101efb359a64"
format_stamp: "Formatted at 2026-09-25 13:21:11 on dist-test-slave-t8d9"
server_key: "6783c27a828c46830018672407e94808"
server_key_iv: "6581c6202806cc693a010fbc62c3e74e"
server_key_version: "encryptionkey@0"
I20260925 13:21:11.462255 18436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:21:11.462906 18451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:11.463045 18436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:11.463129 18436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "8a16eae443b347e0ac8b101efb359a64"
format_stamp: "Formatted at 2026-09-25 13:21:11 on dist-test-slave-t8d9"
server_key: "6783c27a828c46830018672407e94808"
server_key_iv: "6581c6202806cc693a010fbc62c3e74e"
server_key_version: "encryptionkey@0"
I20260925 13:21:11.463191 18436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:11.472299 18436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:11.474391 18436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:11.474522 18436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:11.478044 18436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38149
I20260925 13:21:11.478080 18503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38149 every 8 connection(s)
I20260925 13:21:11.478394 18436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:11.479184 18504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:11.480811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 18436
I20260925 13:21:11.480906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:11.481089 392 external_mini_cluster.cc:1428] Setting key 4da9e850a8a66ca92a324d0e2dc36222
I20260925 13:21:11.481513 18504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64: Bootstrap starting.
I20260925 13:21:11.482218 18504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:11.482518 18504 log.cc:824] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:11.483165 18504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64: No bootstrap required, opened a new log
I20260925 13:21:11.485301 18504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a16eae443b347e0ac8b101efb359a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38149 } }
I20260925 13:21:11.485466 18504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:11.485497 18504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a16eae443b347e0ac8b101efb359a64, State: Initialized, Role: FOLLOWER
I20260925 13:21:11.485658 18504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a16eae443b347e0ac8b101efb359a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38149 } }
I20260925 13:21:11.485734 18504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:11.485769 18504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:11.485811 18504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:11.486389 18504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a16eae443b347e0ac8b101efb359a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38149 } }
I20260925 13:21:11.486488 18504 leader_election.cc:302] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a16eae443b347e0ac8b101efb359a64; no voters:
I20260925 13:21:11.486636 18504 leader_election.cc:288] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:11.486671 18509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:11.486780 18509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [term 1 LEADER]: Becoming Leader. State: Replica: 8a16eae443b347e0ac8b101efb359a64, State: Running, Role: LEADER
I20260925 13:21:11.486861 18509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a16eae443b347e0ac8b101efb359a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38149 } }
I20260925 13:21:11.486898 18504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:11.487246 18509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a16eae443b347e0ac8b101efb359a64. Latest consensus state: current_term: 1 leader_uuid: "8a16eae443b347e0ac8b101efb359a64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a16eae443b347e0ac8b101efb359a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38149 } } }
I20260925 13:21:11.487453 18509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:11.487799 18510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a16eae443b347e0ac8b101efb359a64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a16eae443b347e0ac8b101efb359a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38149 } } }
I20260925 13:21:11.487902 18510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64 [sys.catalog]: This master's current role is: LEADER
Sep 25 13:21:11 dist-test-slave-t8d9 krb5kdc[18423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:11.488092 18518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:11.488636 18518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:11.490315 18518 catalog_manager.cc:1347] Generated new cluster ID: ce46c87864584fa78187decac745cc94
I20260925 13:21:11.490357 18518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:11.492626 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:11.481565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54466 (local address 127.0.98.62:38149)
0925 13:21:11.481718 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:11.481726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:11.481761 (+ 35us) server_negotiation.cc:407] Connection header received
0925 13:21:11.482302 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:11.482313 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:11.482384 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:11.482551 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:11.483139 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:11.484497 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:11.485256 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:11.485898 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:11.486009 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:11.488623 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:11.488628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:11.488628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:11.488641 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:11.491034 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:11.491427 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:11.491431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:11.491431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:11.491545 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:11.491716 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:11.491717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:11.491717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:11.492029 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:11.492186 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:11.492257 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 13:21:11.492408 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260925 13:21:11.513054 18518 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:11.513868 18518 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:11.516243 18518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64: Generated new TSK 0
I20260925 13:21:11.516438 18518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:21:11.566296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38149
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:11.663450 18530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:11.663671 18530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:11.663697 18530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:11.664846 18530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:11.664882 18530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:11.664908 18530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:21:11.666507 18530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:11.666719 18530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:11.666934 18530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:11.671872 18538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:11.671936 18535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:11.671872 18536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:11.672076 18530 server_base.cc:1034] running on GCE node
I20260925 13:21:11.672281 18530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:11.672508 18530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:11.673636 18530 hybrid_clock.cc:648] HybridClock initialized: now 1790342471673623 us; error 24 us; skew 500 ppm
Sep 25 13:21:11 dist-test-slave-t8d9 krb5kdc[18423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:11.675944 18530 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:21:11.676349 18530 webserver.cc:460] Webserver started at http://127.0.98.1:38115/ using document root <none> and password file <none>
I20260925 13:21:11.676565 18530 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:11.676613 18530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:11.676730 18530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:11.677791 18530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "7025e7cef8e346f0b59dde9a50990cf0"
format_stamp: "Formatted at 2026-09-25 13:21:11 on dist-test-slave-t8d9"
server_key: "11f1c2521a3435482b45cd68e6d7d100"
server_key_iv: "06cdc93b4650e0905fa5728dac955ad6"
server_key_version: "encryptionkey@0"
I20260925 13:21:11.678109 18530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7025e7cef8e346f0b59dde9a50990cf0"
format_stamp: "Formatted at 2026-09-25 13:21:11 on dist-test-slave-t8d9"
server_key: "11f1c2521a3435482b45cd68e6d7d100"
server_key_iv: "06cdc93b4650e0905fa5728dac955ad6"
server_key_version: "encryptionkey@0"
I20260925 13:21:11.679280 18530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:21:11.680032 18545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:11.680197 18530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:21:11.680274 18530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "7025e7cef8e346f0b59dde9a50990cf0"
format_stamp: "Formatted at 2026-09-25 13:21:11 on dist-test-slave-t8d9"
server_key: "11f1c2521a3435482b45cd68e6d7d100"
server_key_iv: "06cdc93b4650e0905fa5728dac955ad6"
server_key_version: "encryptionkey@0"
I20260925 13:21:11.680334 18530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:11.691710 18530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:11.693702 18530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:11.693847 18530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:11.694072 18530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:11.694383 18530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:11.694415 18530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:11.694446 18530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:11.694479 18530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:11.699895 18530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44097
I20260925 13:21:11.699971 18658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44097 every 8 connection(s)
I20260925 13:21:11.700251 18530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:21:11.701402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 18530
I20260925 13:21:11.701500 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:21:11.701719 392 external_mini_cluster.cc:1428] Setting key 3bdbe878301e1f62016fe742ccfdfb2a
Sep 25 13:21:11 dist-test-slave-t8d9 krb5kdc[18423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 13:21:11.710170 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:11.700984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45879 (local address 127.0.98.62:38149)
0925 13:21:11.701050 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:11.701052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:11.701216 (+ 164us) server_negotiation.cc:407] Connection header received
0925 13:21:11.702283 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:11.702285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:11.702298 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:11.702345 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:11.702976 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:11.703598 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:11.704658 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:11.704948 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:11.705012 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:11.707370 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:11.707372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:11.707372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:11.707381 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:11.708824 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:11.709194 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:11.709195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:11.709196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:11.709251 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:11.709516 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:11.709517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:11.709517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:11.709685 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:11.709745 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:11.710033 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 13:21:11.710102 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:21:11.710355 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:11.700936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38149 (local address 127.0.98.1:45879)
0925 13:21:11.701099 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:11.701136 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:11.702182 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:11.702420 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:11.702424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:11.702688 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:11.702883 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:11.702888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:11.703702 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:11.703703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:11.704572 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:11.704575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:11.705016 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:11.705016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:11.705114 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:11.705693 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:11.705701 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:11.707272 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:11.708899 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:11.708905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:11.708906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:11.709123 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:11.709336 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:11.709337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:11.709337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:11.709452 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:11.709746 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:11.709751 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:11.709920 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 13:21:11.710009 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 13:21:11.710141 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260925 13:21:11.710878 18659 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38149
I20260925 13:21:11.710971 18659 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:11.711144 18659 heartbeater.cc:511] Master 127.0.98.62:38149 requested a full tablet report, sending...
I20260925 13:21:11.711714 18468 ts_manager.cc:194] Registered new tserver with Master: 7025e7cef8e346f0b59dde9a50990cf0 (127.0.98.1:44097)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 13:21:11.712431 18468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:21:11.759761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38149
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:11.838835 18664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:11.839114 18664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:11.839150 18664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:11.840358 18664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:11.840409 18664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:11.840446 18664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:21:11.842039 18664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:11.842288 18664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:11.842515 18664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:11.847333 18669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:11.847328 18672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:11.847328 18670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:11.847750 18664 server_base.cc:1034] running on GCE node
I20260925 13:21:11.847941 18664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:11.848169 18664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:11.849323 18664 hybrid_clock.cc:648] HybridClock initialized: now 1790342471849307 us; error 28 us; skew 500 ppm
Sep 25 13:21:11 dist-test-slave-t8d9 krb5kdc[18423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:11.851662 18664 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:21:11.852089 18664 webserver.cc:460] Webserver started at http://127.0.98.2:45039/ using document root <none> and password file <none>
I20260925 13:21:11.852280 18664 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:11.852319 18664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:11.852408 18664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:11.853384 18664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "a3c2742bc75a4450bb7c25a16363f238"
format_stamp: "Formatted at 2026-09-25 13:21:11 on dist-test-slave-t8d9"
server_key: "1adc7ab000c666941ae6ada84586f81b"
server_key_iv: "eced9297c55b25fc289ccc070590da7e"
server_key_version: "encryptionkey@0"
I20260925 13:21:11.853710 18664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a3c2742bc75a4450bb7c25a16363f238"
format_stamp: "Formatted at 2026-09-25 13:21:11 on dist-test-slave-t8d9"
server_key: "1adc7ab000c666941ae6ada84586f81b"
server_key_iv: "eced9297c55b25fc289ccc070590da7e"
server_key_version: "encryptionkey@0"
I20260925 13:21:11.854878 18664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:21:11.855506 18679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:11.855652 18664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:21:11.855729 18664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "a3c2742bc75a4450bb7c25a16363f238"
format_stamp: "Formatted at 2026-09-25 13:21:11 on dist-test-slave-t8d9"
server_key: "1adc7ab000c666941ae6ada84586f81b"
server_key_iv: "eced9297c55b25fc289ccc070590da7e"
server_key_version: "encryptionkey@0"
I20260925 13:21:11.855786 18664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:11.870450 18664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:11.872253 18664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:11.872393 18664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:11.872617 18664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:11.872929 18664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:11.872964 18664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:11.872993 18664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:11.873016 18664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:11.878588 18664 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42459
I20260925 13:21:11.878638 18792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42459 every 8 connection(s)
I20260925 13:21:11.878945 18664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:21:11.884200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 18664
I20260925 13:21:11.884301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:21:11.884464 392 external_mini_cluster.cc:1428] Setting key 30f6509a2aec4cbe30cc87826facd231
Sep 25 13:21:11 dist-test-slave-t8d9 krb5kdc[18423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:11.888023 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:11.879600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52359 (local address 127.0.98.62:38149)
0925 13:21:11.879649 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:11.879651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:11.879848 (+ 197us) server_negotiation.cc:407] Connection header received
0925 13:21:11.880758 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:11.880759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:11.880771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:11.880837 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:11.881441 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:11.882009 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:11.882982 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:11.883173 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:11.883224 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:11.885314 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:11.885316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:11.885316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:11.885324 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:11.886694 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:11.887125 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:11.887126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:11.887126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:11.887173 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:11.887410 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:11.887411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:11.887411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:11.887507 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:11.887564 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:11.887859 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 13:21:11.887935 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 13:21:11.888247 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:11.879568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38149 (local address 127.0.98.2:52359)
0925 13:21:11.879730 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:11.879772 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:11.880646 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:11.880914 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:11.880917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:11.881163 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:11.881366 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:11.881370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:11.882075 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:11.882075 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:11.882890 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:11.882892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:11.883233 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:11.883234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:11.883342 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:11.883754 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:11.883761 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:11.885238 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:11.886762 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:11.886768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:11.886771 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:11.887055 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:11.887218 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:11.887219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:11.887219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:11.887346 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:11.887582 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:11.887588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:11.887734 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 13:21:11.887825 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 13:21:11.887993 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260925 13:21:11.888701 18793 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38149
I20260925 13:21:11.888792 18793 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:11.888981 18793 heartbeater.cc:511] Master 127.0.98.62:38149 requested a full tablet report, sending...
I20260925 13:21:11.889362 18468 ts_manager.cc:194] Registered new tserver with Master: a3c2742bc75a4450bb7c25a16363f238 (127.0.98.2:42459)
I20260925 13:21:11.889873 18468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52359
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:21:11.915913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38149
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:11.995141 18798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:11.995334 18798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:11.995355 18798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:11.996443 18798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:11.996483 18798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:11.996510 18798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:21:11.997979 18798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:11.998165 18798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:11.998340 18798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:12.002996 18803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:12.003087 18798 server_base.cc:1034] running on GCE node
W20260925 13:21:12.003003 18804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:12.003211 18806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:12.003422 18798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:12.003651 18798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:12.004786 18798 hybrid_clock.cc:648] HybridClock initialized: now 1790342472004774 us; error 31 us; skew 500 ppm
Sep 25 13:21:12 dist-test-slave-t8d9 krb5kdc[18423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:12.007267 18798 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:21:12.007654 18798 webserver.cc:460] Webserver started at http://127.0.98.3:41967/ using document root <none> and password file <none>
I20260925 13:21:12.007841 18798 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:12.007879 18798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:12.007972 18798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:12.008960 18798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c91697d6fb44764a1316d3f79a235d5"
format_stamp: "Formatted at 2026-09-25 13:21:12 on dist-test-slave-t8d9"
server_key: "a387eb6b4c293ce68ad18d91af6c91f2"
server_key_iv: "b810018ed16f0e80960eb973d87a665e"
server_key_version: "encryptionkey@0"
I20260925 13:21:12.009248 18798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c91697d6fb44764a1316d3f79a235d5"
format_stamp: "Formatted at 2026-09-25 13:21:12 on dist-test-slave-t8d9"
server_key: "a387eb6b4c293ce68ad18d91af6c91f2"
server_key_iv: "b810018ed16f0e80960eb973d87a665e"
server_key_version: "encryptionkey@0"
I20260925 13:21:12.010521 18798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:21:12.011250 18813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:12.011459 18798 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:21:12.011536 18798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "9c91697d6fb44764a1316d3f79a235d5"
format_stamp: "Formatted at 2026-09-25 13:21:12 on dist-test-slave-t8d9"
server_key: "a387eb6b4c293ce68ad18d91af6c91f2"
server_key_iv: "b810018ed16f0e80960eb973d87a665e"
server_key_version: "encryptionkey@0"
I20260925 13:21:12.011597 18798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:12.020773 18798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:12.022755 18798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:12.022881 18798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:12.023078 18798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:12.023339 18798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:12.023368 18798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:12.023396 18798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:12.023412 18798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:12.028733 18798 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36803
I20260925 13:21:12.028798 18926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36803 every 8 connection(s)
I20260925 13:21:12.029102 18798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:21:12.030265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 18798
I20260925 13:21:12.030350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:21:12.030524 392 external_mini_cluster.cc:1428] Setting key 89adc141660316cca0fba7bb8546bbd8
Sep 25 13:21:12 dist-test-slave-t8d9 krb5kdc[18423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:12.037984 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.029863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49399 (local address 127.0.98.62:38149)
0925 13:21:12.029913 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.029916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.030145 (+ 229us) server_negotiation.cc:407] Connection header received
0925 13:21:12.031285 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.031286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.031298 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.031344 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:12.031968 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.032460 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.033400 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.033634 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.033680 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.035631 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:12.035633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:12.035633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:12.035640 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:12.036730 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:12.037068 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:12.037069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:12.037069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:12.037117 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:12.037337 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:12.037338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:12.037338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:12.037473 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:12.037594 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.037822 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.037900 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:21:12.038082 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.029840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38149 (local address 127.0.98.3:49399)
0925 13:21:12.030015 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.030056 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:12.031186 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:12.031432 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:12.031435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:12.031696 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:12.031887 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.031891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.032524 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.032524 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.033337 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.033339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.033685 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.033686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.033787 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.034255 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:12.034263 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:12.035543 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:12.036792 (+ 1249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:12.036798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:12.036799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:12.037012 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:12.037162 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:12.037162 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:12.037163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:12.037258 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:12.037560 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:12.037565 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:12.037685 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 13:21:12.037769 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 13:21:12.037888 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260925 13:21:12.038514 18927 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38149
I20260925 13:21:12.038606 18927 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:12.038776 18927 heartbeater.cc:511] Master 127.0.98.62:38149 requested a full tablet report, sending...
I20260925 13:21:12.039163 18468 ts_manager.cc:194] Registered new tserver with Master: 9c91697d6fb44764a1316d3f79a235d5 (127.0.98.3:36803)
I20260925 13:21:12.039563 18468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49399
I20260925 13:21:12.042169 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:21:12.049002 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.043432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.0.98.62:38149)
0925 13:21:12.043468 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.043469 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.043577 (+ 108us) server_negotiation.cc:407] Connection header received
0925 13:21:12.043616 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.043617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.043625 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.043672 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:12.044244 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.044715 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.045452 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.045693 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.045741 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.046999 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:12.047001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:12.047001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:12.047007 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:12.048064 (+ 1057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:12.048397 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:12.048398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:12.048398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:12.048445 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:12.048590 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:12.048590 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:12.048591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:12.048708 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:12.048787 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.048870 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.048934 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 13:21:12.050496 18468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:21:12.052093 18468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:21:12.058472 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.054285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36803 (local address 127.0.0.1:42140)
0925 13:21:12.054449 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.054490 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:12.054554 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:12.054846 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:12.054848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:12.054850 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:12.054924 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.054926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.056300 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.056300 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.057356 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.057358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.058200 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.058201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.058281 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.058284 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:12.058330 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 13:21:12.058370 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260925 13:21:12.058480 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.054310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42459 (local address 127.0.0.1:58530)
0925 13:21:12.054612 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.054622 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:12.054674 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:12.055156 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:12.055157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:12.055159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:12.055197 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.055198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.056425 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.056426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.057278 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.057280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.058199 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.058200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.058293 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.058296 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:12.058339 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 13:21:12.058374 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":23,"thread_start_us":77,"threads_started":1}
I20260925 13:21:12.058549 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.054432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.0.98.2:42459)
0925 13:21:12.054998 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.055003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.055012 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:21:12.055047 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.055049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.055063 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.055259 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:12.055336 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.056332 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.057349 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.058099 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.058226 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.058286 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.058422 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.058474 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":95,"threads_started":1}
I20260925 13:21:12.058585 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.054488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44097 (local address 127.0.0.1:41738)
0925 13:21:12.054698 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.054707 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:12.054744 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:12.055127 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:12.055127 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:12.055130 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:12.055182 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.055184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.056041 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.056042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.057326 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.057329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.058109 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.058110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.058184 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.058198 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 13:21:12.058263 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 13:21:12.058320 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260925 13:21:12.058869 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.054450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42140 (local address 127.0.98.3:36803)
0925 13:21:12.054701 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.054708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.054719 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:21:12.054757 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.054759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.054776 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.054845 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:12.055068 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.056190 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.057413 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.058099 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.058705 (+ 606us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.058755 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.058775 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.058809 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":76,"threads_started":1}
I20260925 13:21:12.059283 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.054732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.0.98.1:44097)
0925 13:21:12.054881 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.054887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.054896 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:21:12.054932 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.054934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.054946 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.055030 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:12.055256 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.055936 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.057425 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.058034 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.058100 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.058163 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.058422 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.058498 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260925 13:21:12.059545 18861 tablet_service.cc:1467] Processing CreateTablet for tablet 188747d98f5449d7902d603bc74ebe18 (DEFAULT_TABLE table=test-table [id=bd7f80726edc4162a4c7f96d4676cf8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:12.059639 18727 tablet_service.cc:1467] Processing CreateTablet for tablet 188747d98f5449d7902d603bc74ebe18 (DEFAULT_TABLE table=test-table [id=bd7f80726edc4162a4c7f96d4676cf8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:12.059849 18861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188747d98f5449d7902d603bc74ebe18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:12.059898 18727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188747d98f5449d7902d603bc74ebe18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:12.062222 18593 tablet_service.cc:1467] Processing CreateTablet for tablet 188747d98f5449d7902d603bc74ebe18 (DEFAULT_TABLE table=test-table [id=bd7f80726edc4162a4c7f96d4676cf8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:12.062541 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188747d98f5449d7902d603bc74ebe18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:12.063016 18945 tablet_bootstrap.cc:492] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238: Bootstrap starting.
I20260925 13:21:12.063656 18945 tablet_bootstrap.cc:654] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:12.064190 18945 log.cc:824] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:12.064913 18945 tablet_bootstrap.cc:492] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238: No bootstrap required, opened a new log
I20260925 13:21:12.064985 18945 ts_tablet_manager.cc:1397] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:21:12.065816 18944 tablet_bootstrap.cc:492] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5: Bootstrap starting.
I20260925 13:21:12.065676 18947 tablet_bootstrap.cc:492] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0: Bootstrap starting.
I20260925 13:21:12.066542 18944 tablet_bootstrap.cc:654] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:12.066558 18947 tablet_bootstrap.cc:654] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:12.066744 18945 raft_consensus.cc:348] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c91697d6fb44764a1316d3f79a235d5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "a3c2742bc75a4450bb7c25a16363f238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42459 } } peers { permanent_uuid: "7025e7cef8e346f0b59dde9a50990cf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } }
I20260925 13:21:12.066897 18945 raft_consensus.cc:374] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:12.066937 18945 raft_consensus.cc:729] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c2742bc75a4450bb7c25a16363f238, State: Initialized, Role: FOLLOWER
I20260925 13:21:12.067060 18947 log.cc:824] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:12.067068 18945 consensus_queue.cc:260] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c91697d6fb44764a1316d3f79a235d5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "a3c2742bc75a4450bb7c25a16363f238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42459 } } peers { permanent_uuid: "7025e7cef8e346f0b59dde9a50990cf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } }
I20260925 13:21:12.067142 18944 log.cc:824] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:12.067330 18945 ts_tablet_manager.cc:1428] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 13:21:12.067463 18793 heartbeater.cc:503] Master 127.0.98.62:38149 was elected leader, sending a full tablet report...
I20260925 13:21:12.068010 18947 tablet_bootstrap.cc:492] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0: No bootstrap required, opened a new log
I20260925 13:21:12.068079 18947 ts_tablet_manager.cc:1397] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:21:12.068122 18944 tablet_bootstrap.cc:492] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5: No bootstrap required, opened a new log
I20260925 13:21:12.068190 18944 ts_tablet_manager.cc:1397] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:21:12.069697 18947 raft_consensus.cc:348] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c91697d6fb44764a1316d3f79a235d5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "a3c2742bc75a4450bb7c25a16363f238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42459 } } peers { permanent_uuid: "7025e7cef8e346f0b59dde9a50990cf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } }
I20260925 13:21:12.069833 18947 raft_consensus.cc:374] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:12.069854 18947 raft_consensus.cc:729] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7025e7cef8e346f0b59dde9a50990cf0, State: Initialized, Role: FOLLOWER
I20260925 13:21:12.069947 18947 consensus_queue.cc:260] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c91697d6fb44764a1316d3f79a235d5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "a3c2742bc75a4450bb7c25a16363f238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42459 } } peers { permanent_uuid: "7025e7cef8e346f0b59dde9a50990cf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } }
I20260925 13:21:12.069988 18944 raft_consensus.cc:348] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c91697d6fb44764a1316d3f79a235d5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "a3c2742bc75a4450bb7c25a16363f238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42459 } } peers { permanent_uuid: "7025e7cef8e346f0b59dde9a50990cf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } }
I20260925 13:21:12.070149 18944 raft_consensus.cc:374] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:12.070188 18944 raft_consensus.cc:729] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c91697d6fb44764a1316d3f79a235d5, State: Initialized, Role: FOLLOWER
I20260925 13:21:12.070187 18947 ts_tablet_manager.cc:1428] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:21:12.070195 18659 heartbeater.cc:503] Master 127.0.98.62:38149 was elected leader, sending a full tablet report...
I20260925 13:21:12.070302 18944 consensus_queue.cc:260] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c91697d6fb44764a1316d3f79a235d5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "a3c2742bc75a4450bb7c25a16363f238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42459 } } peers { permanent_uuid: "7025e7cef8e346f0b59dde9a50990cf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } }
I20260925 13:21:12.070569 18944 ts_tablet_manager.cc:1428] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:21:12.070648 18927 heartbeater.cc:503] Master 127.0.98.62:38149 was elected leader, sending a full tablet report...
W20260925 13:21:12.129488 18794 tablet.cc:2369] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:21:12.133597 18951 raft_consensus.cc:482] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:21:12.133726 18951 raft_consensus.cc:504] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c91697d6fb44764a1316d3f79a235d5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "a3c2742bc75a4450bb7c25a16363f238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42459 } } peers { permanent_uuid: "7025e7cef8e346f0b59dde9a50990cf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } }
I20260925 13:21:12.134109 18951 leader_election.cc:288] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c91697d6fb44764a1316d3f79a235d5 (127.0.98.3:36803), a3c2742bc75a4450bb7c25a16363f238 (127.0.98.2:42459)
I20260925 13:21:12.137864 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.134385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58999 (local address 127.0.98.2:42459)
0925 13:21:12.134434 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.134436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.134564 (+ 128us) server_negotiation.cc:407] Connection header received
0925 13:21:12.134589 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.134590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.134600 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.134649 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:12.134761 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.135475 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.136871 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.137471 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.137576 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.137589 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.137749 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.137799 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:21:12.137864 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.134348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52347 (local address 127.0.98.3:36803)
0925 13:21:12.134393 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.134395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.134403 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:21:12.134433 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.134434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.134442 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.134496 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:12.134661 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.135407 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.136871 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.137477 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.137581 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.137591 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.137749 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.137790 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:21:12.137871 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.134258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36803 (local address 127.0.98.1:52347)
0925 13:21:12.134304 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.134316 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:12.134361 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:12.134514 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:12.134515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:12.134517 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:12.134590 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.134592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.135480 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.135480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.136739 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.136743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.137580 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.137581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.137682 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.137684 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:21:12.137732 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 13:21:12.137777 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":33}
I20260925 13:21:12.137877 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.134293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42459 (local address 127.0.98.1:58999)
0925 13:21:12.134487 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.134496 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:12.134548 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:12.134656 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:12.134657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:12.134658 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:12.134708 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.134709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.135536 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.135536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.136739 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.136743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.137581 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.137581 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.137677 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.137681 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:21:12.137732 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 13:21:12.137777 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":22,"thread_start_us":114,"threads_started":1}
I20260925 13:21:12.138249 18747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188747d98f5449d7902d603bc74ebe18" candidate_uuid: "7025e7cef8e346f0b59dde9a50990cf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c2742bc75a4450bb7c25a16363f238" is_pre_election: true
I20260925 13:21:12.138257 18881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188747d98f5449d7902d603bc74ebe18" candidate_uuid: "7025e7cef8e346f0b59dde9a50990cf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c91697d6fb44764a1316d3f79a235d5" is_pre_election: true
I20260925 13:21:12.138409 18747 raft_consensus.cc:2454] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7025e7cef8e346f0b59dde9a50990cf0 in term 0.
I20260925 13:21:12.138415 18881 raft_consensus.cc:2454] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7025e7cef8e346f0b59dde9a50990cf0 in term 0.
I20260925 13:21:12.138628 18548 leader_election.cc:302] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7025e7cef8e346f0b59dde9a50990cf0, a3c2742bc75a4450bb7c25a16363f238; no voters:
I20260925 13:21:12.138793 18951 raft_consensus.cc:2781] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:21:12.138840 18951 raft_consensus.cc:482] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:21:12.138854 18951 raft_consensus.cc:3037] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:12.139506 18951 raft_consensus.cc:504] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c91697d6fb44764a1316d3f79a235d5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "a3c2742bc75a4450bb7c25a16363f238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42459 } } peers { permanent_uuid: "7025e7cef8e346f0b59dde9a50990cf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } }
I20260925 13:21:12.139637 18951 leader_election.cc:288] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [CANDIDATE]: Term 1 election: Requested vote from peers 9c91697d6fb44764a1316d3f79a235d5 (127.0.98.3:36803), a3c2742bc75a4450bb7c25a16363f238 (127.0.98.2:42459)
I20260925 13:21:12.139792 18881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188747d98f5449d7902d603bc74ebe18" candidate_uuid: "7025e7cef8e346f0b59dde9a50990cf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c91697d6fb44764a1316d3f79a235d5"
I20260925 13:21:12.139819 18747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188747d98f5449d7902d603bc74ebe18" candidate_uuid: "7025e7cef8e346f0b59dde9a50990cf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c2742bc75a4450bb7c25a16363f238"
I20260925 13:21:12.139868 18881 raft_consensus.cc:3037] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:12.139905 18747 raft_consensus.cc:3037] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:12.140525 18881 raft_consensus.cc:2454] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7025e7cef8e346f0b59dde9a50990cf0 in term 1.
I20260925 13:21:12.140677 18747 raft_consensus.cc:2454] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7025e7cef8e346f0b59dde9a50990cf0 in term 1.
I20260925 13:21:12.140702 18546 leader_election.cc:302] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7025e7cef8e346f0b59dde9a50990cf0, 9c91697d6fb44764a1316d3f79a235d5; no voters:
I20260925 13:21:12.140883 18951 raft_consensus.cc:2781] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:12.141040 18951 raft_consensus.cc:686] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [term 1 LEADER]: Becoming Leader. State: Replica: 7025e7cef8e346f0b59dde9a50990cf0, State: Running, Role: LEADER
I20260925 13:21:12.141139 18951 consensus_queue.cc:237] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c91697d6fb44764a1316d3f79a235d5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } } peers { permanent_uuid: "a3c2742bc75a4450bb7c25a16363f238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42459 } } peers { permanent_uuid: "7025e7cef8e346f0b59dde9a50990cf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } }
I20260925 13:21:12.142014 18467 catalog_manager.cc:5495] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7025e7cef8e346f0b59dde9a50990cf0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7025e7cef8e346f0b59dde9a50990cf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c91697d6fb44764a1316d3f79a235d5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3c2742bc75a4450bb7c25a16363f238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7025e7cef8e346f0b59dde9a50990cf0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } health_report { overall_health: HEALTHY } } }
I20260925 13:21:12.153865 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.151886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.0.98.1:44097)
0925 13:21:12.151935 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.151937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.151947 (+ 10us) server_negotiation.cc:407] Connection header received
0925 13:21:12.151980 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.151980 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.151997 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.152053 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:12.152140 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.152591 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.153249 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.153430 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.153471 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.153613 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:12.153698 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:12.153765 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.153784 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.153792 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 13:21:12.155570 18881 raft_consensus.cc:1261] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5 [term 1 FOLLOWER]: Refusing update from remote peer 7025e7cef8e346f0b59dde9a50990cf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:12.155570 18747 raft_consensus.cc:1261] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238 [term 1 FOLLOWER]: Refusing update from remote peer 7025e7cef8e346f0b59dde9a50990cf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:12.155921 18954 consensus_queue.cc:1035] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3c2742bc75a4450bb7c25a16363f238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:21:12.156051 18951 consensus_queue.cc:1035] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c91697d6fb44764a1316d3f79a235d5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:21:12.158727 18958 mvcc.cc:205] Tried to move back new op lower bound from 7333242765946699776 to 7333242765890859008. Current Snapshot: MvccSnapshot[applied={T|T < 7333242765946699776}]
I20260925 13:21:12.159487 18959 mvcc.cc:205] Tried to move back new op lower bound from 7333242765946699776 to 7333242765890859008. Current Snapshot: MvccSnapshot[applied={T|T < 7333242765946699776 or (T in {7333242765946699776})}]
I20260925 13:21:12.160576 18467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:21:12.160660 18467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:21:12.161299 18467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a16eae443b347e0ac8b101efb359a64: Sending DeleteTablet for 3 replicas of tablet 188747d98f5449d7902d603bc74ebe18
I20260925 13:21:12.161608 18861 tablet_service.cc:1514] Processing DeleteTablet for tablet 188747d98f5449d7902d603bc74ebe18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42140
I20260925 13:21:12.161654 18727 tablet_service.cc:1514] Processing DeleteTablet for tablet 188747d98f5449d7902d603bc74ebe18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58530
I20260925 13:21:12.161818 18593 tablet_service.cc:1514] Processing DeleteTablet for tablet 188747d98f5449d7902d603bc74ebe18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41738
I20260925 13:21:12.162002 18969 tablet_replica.cc:330] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0: stopping tablet replica
I20260925 13:21:12.162102 18969 raft_consensus.cc:2229] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:21:12.162192 18969 raft_consensus.cc:2258] T 188747d98f5449d7902d603bc74ebe18 P 7025e7cef8e346f0b59dde9a50990cf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:12.162281 18968 tablet_replica.cc:330] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238: stopping tablet replica
I20260925 13:21:12.162362 18968 raft_consensus.cc:2229] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:21:12.162382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 18530
I20260925 13:21:12.162451 18968 raft_consensus.cc:2258] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:12.162577 18967 tablet_replica.cc:330] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5: stopping tablet replica
I20260925 13:21:12.162645 18967 raft_consensus.cc:2229] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:21:12.162711 18967 raft_consensus.cc:2258] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:12.163106 18967 ts_tablet_manager.cc:1905] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:21:12.163887 18968 ts_tablet_manager.cc:1905] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:21:12.164672 18967 ts_tablet_manager.cc:1918] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:21:12.164738 18967 log.cc:1196] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/188747d98f5449d7902d603bc74ebe18
I20260925 13:21:12.164999 18967 ts_tablet_manager.cc:1939] T 188747d98f5449d7902d603bc74ebe18 P 9c91697d6fb44764a1316d3f79a235d5: Deleting consensus metadata
I20260925 13:21:12.165400 18968 ts_tablet_manager.cc:1918] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:21:12.165480 18968 log.cc:1196] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/188747d98f5449d7902d603bc74ebe18
I20260925 13:21:12.165452 18452 catalog_manager.cc:4841] TS 9c91697d6fb44764a1316d3f79a235d5 (127.0.98.3:36803): tablet 188747d98f5449d7902d603bc74ebe18 (table test-table [id=bd7f80726edc4162a4c7f96d4676cf8c]) successfully deleted
I20260925 13:21:12.165738 18968 ts_tablet_manager.cc:1939] T 188747d98f5449d7902d603bc74ebe18 P a3c2742bc75a4450bb7c25a16363f238: Deleting consensus metadata
I20260925 13:21:12.166085 18454 catalog_manager.cc:4841] TS a3c2742bc75a4450bb7c25a16363f238 (127.0.98.2:42459): tablet 188747d98f5449d7902d603bc74ebe18 (table test-table [id=bd7f80726edc4162a4c7f96d4676cf8c]) successfully deleted
W20260925 13:21:12.167893 18452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44097 (error 108)
I20260925 13:21:12.168205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 18664
I20260925 13:21:12.168359 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.168235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44097 (local address 127.0.0.1:41764)
0925 13:21:12.168286 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.168326 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 13:21:12.168804 18452 catalog_manager.cc:4584] TS 7025e7cef8e346f0b59dde9a50990cf0 (127.0.98.1:44097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 188747d98f5449d7902d603bc74ebe18: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44097: connect: Connection refused (error 111)
I20260925 13:21:12.172703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 18798
I20260925 13:21:12.176782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 18436
2026-09-25T13:21:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1889 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T13:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:12Z Disabled control of system clock
I20260925 13:21:12.202077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33435
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:21:12.282322 18974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:12.282599 18974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:12.282622 18974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:12.283871 18974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:12.283916 18974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:12.283927 18974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:12.283939 18974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:12.283949 18974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:12.285308 18974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33435
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:12.285600 18974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:12.285902 18974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:12.290555 18982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:12.290570 18980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:12.290624 18979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:12.290799 18974 server_base.cc:1034] running on GCE node
I20260925 13:21:12.291088 18974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:12.291404 18974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:12.292572 18974 hybrid_clock.cc:648] HybridClock initialized: now 1790342472292549 us; error 31 us; skew 500 ppm
I20260925 13:21:12.293919 18974 webserver.cc:460] Webserver started at http://127.0.98.62:41197/ using document root <none> and password file <none>
I20260925 13:21:12.294178 18974 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:12.294227 18974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:12.294351 18974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:12.295445 18974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "a940aabb16964c799bcbb47b7df0e48f"
format_stamp: "Formatted at 2026-09-25 13:21:12 on dist-test-slave-t8d9"
server_key: "924d2706d3dde0c8b9f025ba5d2af456"
server_key_iv: "63dba1d1ea13506a6c1a57206ca29e28"
server_key_version: "encryptionkey@0"
I20260925 13:21:12.295794 18974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "a940aabb16964c799bcbb47b7df0e48f"
format_stamp: "Formatted at 2026-09-25 13:21:12 on dist-test-slave-t8d9"
server_key: "924d2706d3dde0c8b9f025ba5d2af456"
server_key_iv: "63dba1d1ea13506a6c1a57206ca29e28"
server_key_version: "encryptionkey@0"
I20260925 13:21:12.297087 18974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:21:12.297802 18988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:12.297945 18974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:12.298033 18974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "a940aabb16964c799bcbb47b7df0e48f"
format_stamp: "Formatted at 2026-09-25 13:21:12 on dist-test-slave-t8d9"
server_key: "924d2706d3dde0c8b9f025ba5d2af456"
server_key_iv: "63dba1d1ea13506a6c1a57206ca29e28"
server_key_version: "encryptionkey@0"
I20260925 13:21:12.298092 18974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:12.312050 18974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:12.312381 18974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:12.312522 18974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:12.316586 18974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33435
I20260925 13:21:12.316679 19040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33435 every 8 connection(s)
I20260925 13:21:12.317008 18974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:12.318053 19041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:12.320333 19041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f: Bootstrap starting.
I20260925 13:21:12.320861 19041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:12.321090 19041 log.cc:824] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:12.321841 19041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f: No bootstrap required, opened a new log
I20260925 13:21:12.323431 19041 raft_consensus.cc:348] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a940aabb16964c799bcbb47b7df0e48f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33435 } }
I20260925 13:21:12.323576 19041 raft_consensus.cc:374] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:12.323597 19041 raft_consensus.cc:729] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a940aabb16964c799bcbb47b7df0e48f, State: Initialized, Role: FOLLOWER
I20260925 13:21:12.323714 19041 consensus_queue.cc:260] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a940aabb16964c799bcbb47b7df0e48f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33435 } }
I20260925 13:21:12.323766 19041 raft_consensus.cc:388] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:12.323786 19041 raft_consensus.cc:482] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:12.323817 19041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:12.324299 19041 raft_consensus.cc:504] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a940aabb16964c799bcbb47b7df0e48f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33435 } }
I20260925 13:21:12.324393 19041 leader_election.cc:302] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a940aabb16964c799bcbb47b7df0e48f; no voters:
I20260925 13:21:12.324520 19041 leader_election.cc:288] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:12.324579 19044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:12.324735 19044 raft_consensus.cc:686] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [term 1 LEADER]: Becoming Leader. State: Replica: a940aabb16964c799bcbb47b7df0e48f, State: Running, Role: LEADER
I20260925 13:21:12.324754 19041 sys_catalog.cc:564] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:12.324833 19044 consensus_queue.cc:237] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a940aabb16964c799bcbb47b7df0e48f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33435 } }
I20260925 13:21:12.325196 19046 sys_catalog.cc:455] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a940aabb16964c799bcbb47b7df0e48f. Latest consensus state: current_term: 1 leader_uuid: "a940aabb16964c799bcbb47b7df0e48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a940aabb16964c799bcbb47b7df0e48f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33435 } } }
I20260925 13:21:12.325284 19046 sys_catalog.cc:458] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:12.325372 19045 sys_catalog.cc:455] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a940aabb16964c799bcbb47b7df0e48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a940aabb16964c799bcbb47b7df0e48f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33435 } } }
I20260925 13:21:12.325476 19045 sys_catalog.cc:458] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:12.325489 19053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:12.326108 19053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:12.326196 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 18974
I20260925 13:21:12.326303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:12.326515 392 external_mini_cluster.cc:1428] Setting key b8670d2cf9f7cae293da0f907700de7c
I20260925 13:21:12.327656 19053 catalog_manager.cc:1347] Generated new cluster ID: 999150ed105a4487ba1165ad9b2f9bb1
I20260925 13:21:12.327711 19053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:12.331979 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.326966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.0.98.62:33435)
0925 13:21:12.327160 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.327169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.327360 (+ 191us) server_negotiation.cc:407] Connection header received
0925 13:21:12.327483 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.327489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.327560 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.327692 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:12.327888 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.328982 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.329891 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.330364 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.330480 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.331202 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:12.331206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:12.331206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:12.331428 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:12.331461 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:12.331473 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:12.331534 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.331624 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.331812 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260925 13:21:12.353322 19053 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:12.353931 19053 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:12.361364 19053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f: Generated new TSK 0
I20260925 13:21:12.361670 19053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:12.367543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33435
--builtin_ntp_servers=127.0.98.20:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:21:12.460253 19065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:12.460482 19065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:12.460505 19065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:12.461769 19065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:12.461814 19065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:12.461840 19065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:21:12.463297 19065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:12.463537 19065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:12.463838 19065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:12.469512 19070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:12.469508 19073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:12.469506 19071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:12.470005 19065 server_base.cc:1034] running on GCE node
I20260925 13:21:12.470199 19065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:12.470418 19065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:12.471566 19065 hybrid_clock.cc:648] HybridClock initialized: now 1790342472471553 us; error 31 us; skew 500 ppm
I20260925 13:21:12.475958 19065 webserver.cc:460] Webserver started at http://127.0.98.1:42885/ using document root <none> and password file <none>
I20260925 13:21:12.476205 19065 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:12.476257 19065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:12.476384 19065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:12.477476 19065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "0c870a404e3647c58ade4fc923842467"
format_stamp: "Formatted at 2026-09-25 13:21:12 on dist-test-slave-t8d9"
server_key: "8ba5024ac8150e8ef4eac76f50f4f9b7"
server_key_iv: "d826a968c5057bea6a2726df4acead07"
server_key_version: "encryptionkey@0"
I20260925 13:21:12.477859 19065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0c870a404e3647c58ade4fc923842467"
format_stamp: "Formatted at 2026-09-25 13:21:12 on dist-test-slave-t8d9"
server_key: "8ba5024ac8150e8ef4eac76f50f4f9b7"
server_key_iv: "d826a968c5057bea6a2726df4acead07"
server_key_version: "encryptionkey@0"
I20260925 13:21:12.479107 19065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:21:12.479832 19079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:12.480020 19065 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:12.480103 19065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "0c870a404e3647c58ade4fc923842467"
format_stamp: "Formatted at 2026-09-25 13:21:12 on dist-test-slave-t8d9"
server_key: "8ba5024ac8150e8ef4eac76f50f4f9b7"
server_key_iv: "d826a968c5057bea6a2726df4acead07"
server_key_version: "encryptionkey@0"
I20260925 13:21:12.480168 19065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:12.501562 19065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:12.501893 19065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:12.502033 19065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:12.502262 19065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:12.502599 19065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:12.502632 19065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:12.502662 19065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:12.502679 19065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:12.507936 19065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46179
I20260925 13:21:12.508014 19192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46179 every 8 connection(s)
I20260925 13:21:12.508311 19065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:21:12.512969 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 19065
I20260925 13:21:12.513065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:21:12.513238 392 external_mini_cluster.cc:1428] Setting key a18f2860e23f24a4dec0ed457aded39d
I20260925 13:21:12.514104 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.509017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45763 (local address 127.0.98.62:33435)
0925 13:21:12.509078 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.509080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.509287 (+ 207us) server_negotiation.cc:407] Connection header received
0925 13:21:12.509872 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.509873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.509887 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.509958 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:12.510636 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.511314 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.512425 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.512620 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.512689 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.513747 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:12.513749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:12.513749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:12.513753 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:12.513767 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:12.513771 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:12.513815 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.513991 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.514024 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 13:21:12.514366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33435
--builtin_ntp_servers=127.0.98.20:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 13:21:12.514400 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.508999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33435 (local address 127.0.98.1:45763)
0925 13:21:12.509171 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.509212 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:12.509753 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:12.510019 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:12.510023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:12.510031 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:12.510543 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.510548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.511438 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.511439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.512290 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.512293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.512729 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.512730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.512865 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.513563 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:21:12.513564 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:12.513603 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:21:12.513644 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:12.513888 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:12.513896 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:12.513899 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:12.514058 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 13:21:12.514176 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260925 13:21:12.514950 19193 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33435
I20260925 13:21:12.515055 19193 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:12.515254 19193 heartbeater.cc:511] Master 127.0.98.62:33435 requested a full tablet report, sending...
I20260925 13:21:12.515805 19005 ts_manager.cc:194] Registered new tserver with Master: 0c870a404e3647c58ade4fc923842467 (127.0.98.1:46179)
I20260925 13:21:12.516438 19005 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:45763
W20260925 13:21:12.594934 19196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:12.595140 19196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:12.595160 19196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:12.596346 19196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:12.596388 19196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:12.596413 19196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:21:12.597783 19196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:12.598024 19196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:12.598322 19196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:12.603161 19201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:12.603176 19202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:12.603161 19204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:12.603711 19196 server_base.cc:1034] running on GCE node
I20260925 13:21:12.603894 19196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:12.604115 19196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:12.605252 19196 hybrid_clock.cc:648] HybridClock initialized: now 1790342472605239 us; error 24 us; skew 500 ppm
I20260925 13:21:12.606514 19196 webserver.cc:460] Webserver started at http://127.0.98.2:41011/ using document root <none> and password file <none>
I20260925 13:21:12.606746 19196 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:12.606797 19196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:12.606922 19196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:12.608235 19196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "54ab40ef614f474ead221674930aefcc"
format_stamp: "Formatted at 2026-09-25 13:21:12 on dist-test-slave-t8d9"
server_key: "bcb9a0b2ede44562f49b86ad060527e4"
server_key_iv: "c7429144a24ef382842e98e7847cbad9"
server_key_version: "encryptionkey@0"
I20260925 13:21:12.608592 19196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "54ab40ef614f474ead221674930aefcc"
format_stamp: "Formatted at 2026-09-25 13:21:12 on dist-test-slave-t8d9"
server_key: "bcb9a0b2ede44562f49b86ad060527e4"
server_key_iv: "c7429144a24ef382842e98e7847cbad9"
server_key_version: "encryptionkey@0"
I20260925 13:21:12.609869 19196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:21:12.610617 19210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:12.610795 19196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:21:12.610878 19196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "54ab40ef614f474ead221674930aefcc"
format_stamp: "Formatted at 2026-09-25 13:21:12 on dist-test-slave-t8d9"
server_key: "bcb9a0b2ede44562f49b86ad060527e4"
server_key_iv: "c7429144a24ef382842e98e7847cbad9"
server_key_version: "encryptionkey@0"
I20260925 13:21:12.610941 19196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:12.620297 19196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:12.620608 19196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:12.620733 19196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:12.620945 19196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:12.621255 19196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:12.621287 19196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:12.621315 19196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:12.621340 19196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:12.626935 19196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37081
I20260925 13:21:12.626991 19323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37081 every 8 connection(s)
I20260925 13:21:12.627303 19196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:21:12.629236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 19196
I20260925 13:21:12.629351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:21:12.629586 392 external_mini_cluster.cc:1428] Setting key 96938a98c7ce6f48deb1ac872c2f0dce
I20260925 13:21:12.630530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33435
--builtin_ntp_servers=127.0.98.20:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 13:21:12.633452 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.627994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33799 (local address 127.0.98.62:33435)
0925 13:21:12.628046 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.628048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.628286 (+ 238us) server_negotiation.cc:407] Connection header received
0925 13:21:12.628797 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.628798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.628806 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.628867 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:12.629835 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.630406 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.631851 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.632158 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.632245 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.633199 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:12.633201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:12.633201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:12.633207 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:12.633222 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:12.633226 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:12.633270 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.633359 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.633389 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 13:21:12.633745 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.627977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33435 (local address 127.0.98.2:33799)
0925 13:21:12.628164 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.628207 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:12.628716 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:12.628952 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:12.628956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:12.628966 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:12.629715 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.629723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.630507 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.630509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.631727 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.631731 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.632286 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.632287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.632433 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.633019 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:21:12.633020 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:12.633065 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:21:12.633097 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:12.633287 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:12.633292 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:12.633295 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:12.633360 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 13:21:12.633484 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260925 13:21:12.634289 19324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33435
I20260925 13:21:12.634395 19324 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:12.634582 19324 heartbeater.cc:511] Master 127.0.98.62:33435 requested a full tablet report, sending...
I20260925 13:21:12.634976 19005 ts_manager.cc:194] Registered new tserver with Master: 54ab40ef614f474ead221674930aefcc (127.0.98.2:37081)
I20260925 13:21:12.635349 19005 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:33799
W20260925 13:21:12.711622 19327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:12.711850 19327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:12.711875 19327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:12.713083 19327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:12.713132 19327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:12.713158 19327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:21:12.714666 19327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:12.714927 19327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:12.715250 19327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:12.720052 19335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:12.720064 19333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:12.720059 19332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:12.720270 19327 server_base.cc:1034] running on GCE node
I20260925 13:21:12.720475 19327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:12.720681 19327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:12.721813 19327 hybrid_clock.cc:648] HybridClock initialized: now 1790342472721804 us; error 29 us; skew 500 ppm
I20260925 13:21:12.722985 19327 webserver.cc:460] Webserver started at http://127.0.98.3:35063/ using document root <none> and password file <none>
I20260925 13:21:12.723208 19327 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:12.723256 19327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:12.723377 19327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:12.724586 19327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "97e91f8cbf26458390bdbcb90d29c5d7"
format_stamp: "Formatted at 2026-09-25 13:21:12 on dist-test-slave-t8d9"
server_key: "b193aa9a2a20ffffbe0578434e0eba43"
server_key_iv: "537acc5552f98509c3b2296d337094e8"
server_key_version: "encryptionkey@0"
I20260925 13:21:12.724982 19327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "97e91f8cbf26458390bdbcb90d29c5d7"
format_stamp: "Formatted at 2026-09-25 13:21:12 on dist-test-slave-t8d9"
server_key: "b193aa9a2a20ffffbe0578434e0eba43"
server_key_iv: "537acc5552f98509c3b2296d337094e8"
server_key_version: "encryptionkey@0"
I20260925 13:21:12.726593 19327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:21:12.729696 19341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:12.729929 19327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 13:21:12.730031 19327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "97e91f8cbf26458390bdbcb90d29c5d7"
format_stamp: "Formatted at 2026-09-25 13:21:12 on dist-test-slave-t8d9"
server_key: "b193aa9a2a20ffffbe0578434e0eba43"
server_key_iv: "537acc5552f98509c3b2296d337094e8"
server_key_version: "encryptionkey@0"
I20260925 13:21:12.730115 19327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:12.745482 19327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:12.745842 19327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:12.745971 19327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:12.746192 19327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:12.746505 19327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:12.746538 19327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:12.746567 19327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:12.746593 19327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:12.751935 19327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46521
I20260925 13:21:12.751998 19454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46521 every 8 connection(s)
I20260925 13:21:12.752291 19327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:21:12.756191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 19327
I20260925 13:21:12.756306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:21:12.756493 392 external_mini_cluster.cc:1428] Setting key 9bb980b0000ad5d5942f526964249069
I20260925 13:21:12.757385 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.752972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46447 (local address 127.0.98.62:33435)
0925 13:21:12.753034 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.753036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.753242 (+ 206us) server_negotiation.cc:407] Connection header received
0925 13:21:12.753758 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.753759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.753768 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.753833 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:12.754455 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.754957 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.755885 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.756096 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.756146 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.757112 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:12.757113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:12.757114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:12.757118 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:12.757136 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:12.757140 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:12.757191 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.757272 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.757297 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:21:12.757699 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.752949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33435 (local address 127.0.98.3:46447)
0925 13:21:12.753129 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.753169 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:12.753683 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:12.753892 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:12.753895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:12.753903 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:12.754384 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.754389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.755027 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.755027 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.755806 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.755807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.756176 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.756177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.756340 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.756965 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:21:12.756966 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:12.757006 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:21:12.757033 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:12.757198 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:12.757203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:12.757206 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:12.757275 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 13:21:12.757383 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260925 13:21:12.758185 19455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33435
I20260925 13:21:12.758270 19455 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:12.758440 19455 heartbeater.cc:511] Master 127.0.98.62:33435 requested a full tablet report, sending...
I20260925 13:21:12.758751 19005 ts_manager.cc:194] Registered new tserver with Master: 97e91f8cbf26458390bdbcb90d29c5d7 (127.0.98.3:46521)
I20260925 13:21:12.759126 19005 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46447
I20260925 13:21:12.768288 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:21:12.772229 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.769635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.0.98.62:33435)
0925 13:21:12.769699 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.769702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.769796 (+ 94us) server_negotiation.cc:407] Connection header received
0925 13:21:12.769867 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.769868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.769880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.769922 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:12.770053 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.770525 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.771212 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.771380 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.771420 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.772004 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:12.772005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:12.772005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:12.772010 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:12.772022 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:12.772026 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:12.772099 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.772136 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.772147 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 13:21:12.773736 19005 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:21:12.775276 19005 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:21:12.781586 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.777442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37081 (local address 127.0.0.1:42186)
0925 13:21:12.777634 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.777681 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:12.777748 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:12.778309 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:12.778310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:12.778312 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:12.778343 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.778344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.779630 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.779631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.780478 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.780480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.781316 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.781317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.781381 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.781386 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:21:12.781415 (+ 29us) client_negotiation.cc:240] Negotiation successful
0925 13:21:12.781488 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":38,"thread_start_us":72,"threads_started":1}
I20260925 13:21:12.781592 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.777867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49974 (local address 127.0.98.3:46521)
0925 13:21:12.778073 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.778076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.778084 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:21:12.778119 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.778121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.778136 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.778203 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:12.778351 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.779120 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.780515 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.781158 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.781222 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.781324 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.781427 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.781470 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":101,"threads_started":1}
I20260925 13:21:12.781757 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.777578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.0.98.2:37081)
0925 13:21:12.777986 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.777990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.777999 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:21:12.778034 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.778036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.778049 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.778125 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:12.778419 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.779532 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.780553 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.781219 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.781461 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.781602 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.781627 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.781669 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":89,"threads_started":1}
I20260925 13:21:12.781822 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.777442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46179 (local address 127.0.0.1:47106)
0925 13:21:12.777640 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.777681 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:12.777747 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:12.778135 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:12.778138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:12.778140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:12.778238 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.778240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.779615 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.779616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.780703 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.780706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.781630 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.781631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.781686 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.781690 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:21:12.781718 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 13:21:12.781757 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":31,"thread_start_us":121,"threads_started":1}
I20260925 13:21:12.782171 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.777678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46521 (local address 127.0.0.1:49974)
0925 13:21:12.777918 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.777928 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:12.777980 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:12.778220 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:12.778221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:12.778224 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:12.778269 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.778271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.779205 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.779206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.780416 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.780418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.781242 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.781243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.781322 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.781336 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 13:21:12.781405 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 13:21:12.782098 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":70,"threads_started":1}
I20260925 13:21:12.782202 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.777773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.0.98.1:46179)
0925 13:21:12.777945 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.777950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.777976 (+ 26us) server_negotiation.cc:407] Connection header received
0925 13:21:12.778020 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.778022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.778039 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.778125 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:12.778304 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.779513 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.780790 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.781510 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.781887 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.781976 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.781998 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.782031 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260925 13:21:12.782801 19258 tablet_service.cc:1467] Processing CreateTablet for tablet 8b83bd180a9c4e9ca125479f415465c1 (DEFAULT_TABLE table=test-table [id=45090be1a1854ac9ab131d658f5d617c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:12.782856 19127 tablet_service.cc:1467] Processing CreateTablet for tablet 8b83bd180a9c4e9ca125479f415465c1 (DEFAULT_TABLE table=test-table [id=45090be1a1854ac9ab131d658f5d617c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:12.782869 19389 tablet_service.cc:1467] Processing CreateTablet for tablet 8b83bd180a9c4e9ca125479f415465c1 (DEFAULT_TABLE table=test-table [id=45090be1a1854ac9ab131d658f5d617c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:12.783190 19258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b83bd180a9c4e9ca125479f415465c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:12.783198 19127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b83bd180a9c4e9ca125479f415465c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:12.783198 19389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b83bd180a9c4e9ca125479f415465c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:12.785732 19469 tablet_bootstrap.cc:492] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc: Bootstrap starting.
I20260925 13:21:12.786389 19469 tablet_bootstrap.cc:654] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:12.786435 19471 tablet_bootstrap.cc:492] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7: Bootstrap starting.
I20260925 13:21:12.786716 19469 log.cc:824] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:12.787053 19471 tablet_bootstrap.cc:654] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:12.787258 19470 tablet_bootstrap.cc:492] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467: Bootstrap starting.
I20260925 13:21:12.787364 19471 log.cc:824] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:12.787441 19469 tablet_bootstrap.cc:492] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc: No bootstrap required, opened a new log
I20260925 13:21:12.787509 19469 ts_tablet_manager.cc:1397] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:21:12.787825 19470 tablet_bootstrap.cc:654] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:12.788107 19471 tablet_bootstrap.cc:492] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7: No bootstrap required, opened a new log
I20260925 13:21:12.788125 19470 log.cc:824] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:12.788173 19471 ts_tablet_manager.cc:1397] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:21:12.788878 19470 tablet_bootstrap.cc:492] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467: No bootstrap required, opened a new log
I20260925 13:21:12.788950 19470 ts_tablet_manager.cc:1397] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:21:12.789678 19469 raft_consensus.cc:348] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c870a404e3647c58ade4fc923842467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46179 } } peers { permanent_uuid: "54ab40ef614f474ead221674930aefcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "97e91f8cbf26458390bdbcb90d29c5d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46521 } }
I20260925 13:21:12.789845 19469 raft_consensus.cc:374] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:12.789883 19469 raft_consensus.cc:729] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ab40ef614f474ead221674930aefcc, State: Initialized, Role: FOLLOWER
I20260925 13:21:12.789999 19469 consensus_queue.cc:260] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c870a404e3647c58ade4fc923842467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46179 } } peers { permanent_uuid: "54ab40ef614f474ead221674930aefcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "97e91f8cbf26458390bdbcb90d29c5d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46521 } }
I20260925 13:21:12.790241 19469 ts_tablet_manager.cc:1428] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:21:12.790385 19324 heartbeater.cc:503] Master 127.0.98.62:33435 was elected leader, sending a full tablet report...
I20260925 13:21:12.790561 19471 raft_consensus.cc:348] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c870a404e3647c58ade4fc923842467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46179 } } peers { permanent_uuid: "54ab40ef614f474ead221674930aefcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "97e91f8cbf26458390bdbcb90d29c5d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46521 } }
I20260925 13:21:12.790735 19471 raft_consensus.cc:374] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:12.790773 19471 raft_consensus.cc:729] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e91f8cbf26458390bdbcb90d29c5d7, State: Initialized, Role: FOLLOWER
I20260925 13:21:12.790879 19471 consensus_queue.cc:260] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c870a404e3647c58ade4fc923842467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46179 } } peers { permanent_uuid: "54ab40ef614f474ead221674930aefcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "97e91f8cbf26458390bdbcb90d29c5d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46521 } }
I20260925 13:21:12.791118 19471 ts_tablet_manager.cc:1428] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:21:12.791245 19455 heartbeater.cc:503] Master 127.0.98.62:33435 was elected leader, sending a full tablet report...
I20260925 13:21:12.791363 19470 raft_consensus.cc:348] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c870a404e3647c58ade4fc923842467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46179 } } peers { permanent_uuid: "54ab40ef614f474ead221674930aefcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "97e91f8cbf26458390bdbcb90d29c5d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46521 } }
I20260925 13:21:12.791503 19470 raft_consensus.cc:374] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:12.791544 19470 raft_consensus.cc:729] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c870a404e3647c58ade4fc923842467, State: Initialized, Role: FOLLOWER
I20260925 13:21:12.791647 19470 consensus_queue.cc:260] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c870a404e3647c58ade4fc923842467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46179 } } peers { permanent_uuid: "54ab40ef614f474ead221674930aefcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "97e91f8cbf26458390bdbcb90d29c5d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46521 } }
I20260925 13:21:12.791894 19470 ts_tablet_manager.cc:1428] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:21:12.791945 19193 heartbeater.cc:503] Master 127.0.98.62:33435 was elected leader, sending a full tablet report...
W20260925 13:21:12.877851 19325 tablet.cc:2369] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:21:12.913595 19476 raft_consensus.cc:482] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:21:12.913738 19476 raft_consensus.cc:504] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c870a404e3647c58ade4fc923842467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46179 } } peers { permanent_uuid: "54ab40ef614f474ead221674930aefcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "97e91f8cbf26458390bdbcb90d29c5d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46521 } }
I20260925 13:21:12.914135 19476 leader_election.cc:288] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c870a404e3647c58ade4fc923842467 (127.0.98.1:46179), 54ab40ef614f474ead221674930aefcc (127.0.98.2:37081)
I20260925 13:21:12.917038 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.914414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37327 (local address 127.0.98.2:37081)
0925 13:21:12.914471 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.914472 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.914632 (+ 160us) server_negotiation.cc:407] Connection header received
0925 13:21:12.914694 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.914696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.914708 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.914762 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:12.914917 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.915420 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.916334 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.916696 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.916755 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.916766 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.916932 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.916970 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:21:12.917045 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.914346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37081 (local address 127.0.98.3:37327)
0925 13:21:12.914558 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.914569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:12.914626 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:12.914775 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:12.914775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:12.914777 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:12.914841 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.914843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.915502 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.915503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.916252 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.916254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.916773 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.916775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.916868 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.916871 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:12.916917 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 13:21:12.916959 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":54,"thread_start_us":95,"threads_started":1}
I20260925 13:21:12.917097 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.914422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56617 (local address 127.0.98.1:46179)
0925 13:21:12.914463 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.914465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.914477 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:21:12.914507 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.914508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.914520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.914602 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:12.914779 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.915383 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.916336 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.916744 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.916841 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.916853 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.916995 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.917035 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:21:12.917115 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.914339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46179 (local address 127.0.98.3:56617)
0925 13:21:12.914388 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.914400 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:12.914444 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:12.914607 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:12.914608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:12.914610 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:12.914685 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.914688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.915471 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.915472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.916243 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:12.916245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.916869 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.916870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:12.916933 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.916935 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 13:21:12.916980 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 13:21:12.917021 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":23}
I20260925 13:21:12.917400 19278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b83bd180a9c4e9ca125479f415465c1" candidate_uuid: "97e91f8cbf26458390bdbcb90d29c5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ab40ef614f474ead221674930aefcc" is_pre_election: true
I20260925 13:21:12.917399 19147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b83bd180a9c4e9ca125479f415465c1" candidate_uuid: "97e91f8cbf26458390bdbcb90d29c5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c870a404e3647c58ade4fc923842467" is_pre_election: true
I20260925 13:21:12.917569 19147 raft_consensus.cc:2454] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97e91f8cbf26458390bdbcb90d29c5d7 in term 0.
I20260925 13:21:12.917570 19278 raft_consensus.cc:2454] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97e91f8cbf26458390bdbcb90d29c5d7 in term 0.
I20260925 13:21:12.917795 19344 leader_election.cc:302] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54ab40ef614f474ead221674930aefcc, 97e91f8cbf26458390bdbcb90d29c5d7; no voters:
I20260925 13:21:12.917917 19476 raft_consensus.cc:2781] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:21:12.917990 19476 raft_consensus.cc:482] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:21:12.918006 19476 raft_consensus.cc:3037] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:12.918676 19476 raft_consensus.cc:504] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c870a404e3647c58ade4fc923842467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46179 } } peers { permanent_uuid: "54ab40ef614f474ead221674930aefcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "97e91f8cbf26458390bdbcb90d29c5d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46521 } }
I20260925 13:21:12.918799 19476 leader_election.cc:288] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [CANDIDATE]: Term 1 election: Requested vote from peers 0c870a404e3647c58ade4fc923842467 (127.0.98.1:46179), 54ab40ef614f474ead221674930aefcc (127.0.98.2:37081)
I20260925 13:21:12.918951 19278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b83bd180a9c4e9ca125479f415465c1" candidate_uuid: "97e91f8cbf26458390bdbcb90d29c5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ab40ef614f474ead221674930aefcc"
I20260925 13:21:12.918970 19147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b83bd180a9c4e9ca125479f415465c1" candidate_uuid: "97e91f8cbf26458390bdbcb90d29c5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c870a404e3647c58ade4fc923842467"
I20260925 13:21:12.919037 19278 raft_consensus.cc:3037] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:12.919037 19147 raft_consensus.cc:3037] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:12.919935 19278 raft_consensus.cc:2454] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97e91f8cbf26458390bdbcb90d29c5d7 in term 1.
I20260925 13:21:12.919935 19147 raft_consensus.cc:2454] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97e91f8cbf26458390bdbcb90d29c5d7 in term 1.
I20260925 13:21:12.920083 19344 leader_election.cc:302] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54ab40ef614f474ead221674930aefcc, 97e91f8cbf26458390bdbcb90d29c5d7; no voters:
I20260925 13:21:12.920159 19476 raft_consensus.cc:2781] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:12.920276 19476 raft_consensus.cc:686] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [term 1 LEADER]: Becoming Leader. State: Replica: 97e91f8cbf26458390bdbcb90d29c5d7, State: Running, Role: LEADER
I20260925 13:21:12.920370 19476 consensus_queue.cc:237] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c870a404e3647c58ade4fc923842467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46179 } } peers { permanent_uuid: "54ab40ef614f474ead221674930aefcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } } peers { permanent_uuid: "97e91f8cbf26458390bdbcb90d29c5d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46521 } }
I20260925 13:21:12.921176 19004 catalog_manager.cc:5495] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97e91f8cbf26458390bdbcb90d29c5d7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "97e91f8cbf26458390bdbcb90d29c5d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c870a404e3647c58ade4fc923842467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54ab40ef614f474ead221674930aefcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97e91f8cbf26458390bdbcb90d29c5d7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46521 } health_report { overall_health: HEALTHY } } }
I20260925 13:21:12.930686 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.928424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.0.98.3:46521)
0925 13:21:12.928483 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:12.928485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:12.928493 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:21:12.928521 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:12.928522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:12.928538 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:12.928594 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:12.928691 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.929147 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.930019 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:12.930195 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:12.930236 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:12.930371 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:12.930452 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:12.930527 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:12.930572 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 13:21:12.930580 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 13:21:12.932413 19278 raft_consensus.cc:1261] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc [term 1 FOLLOWER]: Refusing update from remote peer 97e91f8cbf26458390bdbcb90d29c5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:12.932415 19147 raft_consensus.cc:1261] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467 [term 1 FOLLOWER]: Refusing update from remote peer 97e91f8cbf26458390bdbcb90d29c5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:12.932732 19476 consensus_queue.cc:1035] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54ab40ef614f474ead221674930aefcc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:21:12.932876 19479 consensus_queue.cc:1035] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c870a404e3647c58ade4fc923842467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:21:12.937692 19003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:21:12.937816 19003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:21:12.938585 19003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a940aabb16964c799bcbb47b7df0e48f: Sending DeleteTablet for 3 replicas of tablet 8b83bd180a9c4e9ca125479f415465c1
I20260925 13:21:12.938968 19389 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b83bd180a9c4e9ca125479f415465c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:12 UTC) from {username='slave'} at 127.0.0.1:49974
I20260925 13:21:12.938930 19258 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b83bd180a9c4e9ca125479f415465c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:12 UTC) from {username='slave'} at 127.0.0.1:42186
I20260925 13:21:12.939019 19127 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b83bd180a9c4e9ca125479f415465c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:12 UTC) from {username='slave'} at 127.0.0.1:47106
I20260925 13:21:12.939232 19492 tablet_replica.cc:330] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc: stopping tablet replica
I20260925 13:21:12.939340 19492 raft_consensus.cc:2229] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:21:12.939450 19492 raft_consensus.cc:2258] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:12.939491 19494 tablet_replica.cc:330] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467: stopping tablet replica
I20260925 13:21:12.939581 19494 raft_consensus.cc:2229] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:21:12.939605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 19065
I20260925 13:21:12.939670 19494 raft_consensus.cc:2258] T 8b83bd180a9c4e9ca125479f415465c1 P 0c870a404e3647c58ade4fc923842467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:12.940610 19493 tablet_replica.cc:330] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7: stopping tablet replica
I20260925 13:21:12.940675 19493 raft_consensus.cc:2229] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:21:12.940757 19493 raft_consensus.cc:2258] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:12.941155 19493 ts_tablet_manager.cc:1905] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:21:12.942745 19492 ts_tablet_manager.cc:1905] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:21:12.942788 19493 ts_tablet_manager.cc:1918] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:21:12.942845 19493 log.cc:1196] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/8b83bd180a9c4e9ca125479f415465c1
I20260925 13:21:12.943071 19493 ts_tablet_manager.cc:1939] T 8b83bd180a9c4e9ca125479f415465c1 P 97e91f8cbf26458390bdbcb90d29c5d7: Deleting consensus metadata
I20260925 13:21:12.943444 18991 catalog_manager.cc:4841] TS 97e91f8cbf26458390bdbcb90d29c5d7 (127.0.98.3:46521): tablet 8b83bd180a9c4e9ca125479f415465c1 (table test-table [id=45090be1a1854ac9ab131d658f5d617c]) successfully deleted
I20260925 13:21:12.944267 19492 ts_tablet_manager.cc:1918] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:21:12.944324 19492 log.cc:1196] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/8b83bd180a9c4e9ca125479f415465c1
I20260925 13:21:12.944479 19492 ts_tablet_manager.cc:1939] T 8b83bd180a9c4e9ca125479f415465c1 P 54ab40ef614f474ead221674930aefcc: Deleting consensus metadata
I20260925 13:21:12.945171 18991 catalog_manager.cc:4841] TS 54ab40ef614f474ead221674930aefcc (127.0.98.2:37081): tablet 8b83bd180a9c4e9ca125479f415465c1 (table test-table [id=45090be1a1854ac9ab131d658f5d617c]) successfully deleted
W20260925 13:21:12.947346 18989 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46179 (error 108)
I20260925 13:21:12.947808 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:12.947692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46179 (local address 127.0.0.1:47110)
0925 13:21:12.947740 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:12.947778 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 13:21:12.947916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 19196
W20260925 13:21:12.948004 18989 catalog_manager.cc:4584] TS 0c870a404e3647c58ade4fc923842467 (127.0.98.1:46179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b83bd180a9c4e9ca125479f415465c1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46179: connect: Connection refused (error 111)
I20260925 13:21:12.954270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 19327
I20260925 13:21:12.958338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 18974
2026-09-25T13:21:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (781 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:13 dist-test-slave-t8d9 krb5kdc[19498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:13 dist-test-slave-t8d9 krb5kdc[19498](info): set up 2 sockets
Sep 25 13:21:13 dist-test-slave-t8d9 krb5kdc[19498](info): commencing operation
krb5kdc: starting...
W20260925 13:21:13.917229 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:13 dist-test-slave-t8d9 krb5kdc[19498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:21:14.032274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41613
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:14.111454 19511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:14.111683 19511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:14.111706 19511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:14.112907 19511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:14.112953 19511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:14.112965 19511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:14.112977 19511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:14.112987 19511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:14.114534 19511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:14.114786 19511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:14.115008 19511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:14.119798 19519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:14.119781 19516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:14.119776 19517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:14.120158 19511 server_base.cc:1034] running on GCE node
I20260925 13:21:14.120374 19511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:14.120648 19511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:14.121783 19511 hybrid_clock.cc:648] HybridClock initialized: now 1790342474121764 us; error 33 us; skew 500 ppm
Sep 25 13:21:14 dist-test-slave-t8d9 krb5kdc[19498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:14.124256 19511 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:21:14.124691 19511 webserver.cc:460] Webserver started at http://127.0.98.62:35455/ using document root <none> and password file <none>
I20260925 13:21:14.124902 19511 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:14.124945 19511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:14.125066 19511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:14.126179 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "3434a146b3d64b6bba937ef15221620b"
format_stamp: "Formatted at 2026-09-25 13:21:14 on dist-test-slave-t8d9"
server_key: "4e26fce0ea62e7c30b3a3d8fdfb7b064"
server_key_iv: "8e8b886042c15268fc0468882d748631"
server_key_version: "encryptionkey@0"
I20260925 13:21:14.126508 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "3434a146b3d64b6bba937ef15221620b"
format_stamp: "Formatted at 2026-09-25 13:21:14 on dist-test-slave-t8d9"
server_key: "4e26fce0ea62e7c30b3a3d8fdfb7b064"
server_key_iv: "8e8b886042c15268fc0468882d748631"
server_key_version: "encryptionkey@0"
I20260925 13:21:14.127686 19511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:21:14.128345 19526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:14.128480 19511 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:14.128551 19511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "3434a146b3d64b6bba937ef15221620b"
format_stamp: "Formatted at 2026-09-25 13:21:14 on dist-test-slave-t8d9"
server_key: "4e26fce0ea62e7c30b3a3d8fdfb7b064"
server_key_iv: "8e8b886042c15268fc0468882d748631"
server_key_version: "encryptionkey@0"
I20260925 13:21:14.128614 19511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:14.146090 19511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:14.147848 19511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:14.147960 19511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:14.151881 19511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41613
I20260925 13:21:14.151925 19578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41613 every 8 connection(s)
I20260925 13:21:14.152266 19511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:14.153110 19579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:14.155426 19579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b: Bootstrap starting.
I20260925 13:21:14.155973 19579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:14.156195 19579 log.cc:824] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:14.156898 19579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b: No bootstrap required, opened a new log
I20260925 13:21:14.157088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 19511
I20260925 13:21:14.157183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:14.157359 392 external_mini_cluster.cc:1428] Setting key 640cd6cac048cde9211017a5f59d9a4e
I20260925 13:21:14.159015 19579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3434a146b3d64b6bba937ef15221620b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } }
I20260925 13:21:14.159200 19579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:14.159245 19579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3434a146b3d64b6bba937ef15221620b, State: Initialized, Role: FOLLOWER
I20260925 13:21:14.159380 19579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3434a146b3d64b6bba937ef15221620b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } }
I20260925 13:21:14.159463 19579 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:14.159500 19579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:14.159564 19579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:14.160099 19579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3434a146b3d64b6bba937ef15221620b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } }
I20260925 13:21:14.160190 19579 leader_election.cc:302] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3434a146b3d64b6bba937ef15221620b; no voters:
I20260925 13:21:14.160347 19579 leader_election.cc:288] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:14.160368 19584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:14.160471 19584 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [term 1 LEADER]: Becoming Leader. State: Replica: 3434a146b3d64b6bba937ef15221620b, State: Running, Role: LEADER
I20260925 13:21:14.160552 19584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3434a146b3d64b6bba937ef15221620b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } }
I20260925 13:21:14.160627 19579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:14.160866 19586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3434a146b3d64b6bba937ef15221620b. Latest consensus state: current_term: 1 leader_uuid: "3434a146b3d64b6bba937ef15221620b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3434a146b3d64b6bba937ef15221620b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } }
I20260925 13:21:14.160954 19586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:14.160861 19585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3434a146b3d64b6bba937ef15221620b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3434a146b3d64b6bba937ef15221620b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41613 } } }
I20260925 13:21:14.161103 19585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:14.161177 19588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:14.161715 19588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 13:21:14 dist-test-slave-t8d9 krb5kdc[19498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:14.166122 19588 catalog_manager.cc:1347] Generated new cluster ID: 04f04373a3a442bbbfe85f8ee459a3dc
I20260925 13:21:14.166179 19588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:14.169106 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.157872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56410 (local address 127.0.98.62:41613)
0925 13:21:14.158036 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:14.158044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:14.158069 (+ 25us) server_negotiation.cc:407] Connection header received
0925 13:21:14.158626 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:14.158633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:14.158682 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:14.158795 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:14.159562 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.161085 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.162200 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.162781 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.162893 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.165037 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:14.165042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:14.165043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:14.165055 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:14.166983 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:14.167412 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:14.167415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:14.167415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:14.167520 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:14.167762 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:14.167762 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:14.167763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:14.168082 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:14.168671 (+ 589us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:14.168743 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 13:21:14.168896 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260925 13:21:14.206624 19588 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:14.207507 19588 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:14.215699 19588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b: Generated new TSK 0
I20260925 13:21:14.215946 19588 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 13:21:14.240854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41613
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:14.324059 19605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:14.324271 19605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:14.324306 19605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:14.325639 19605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:14.325686 19605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:14.325724 19605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:21:14.327338 19605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:14.327605 19605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:14.327821 19605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:14.332677 19610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:14.332747 19613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:14.332682 19611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:14.333063 19605 server_base.cc:1034] running on GCE node
I20260925 13:21:14.333278 19605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:14.333485 19605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:14.334647 19605 hybrid_clock.cc:648] HybridClock initialized: now 1790342474334634 us; error 27 us; skew 500 ppm
Sep 25 13:21:14 dist-test-slave-t8d9 krb5kdc[19498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:14.337230 19605 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 13:21:14.337733 19605 webserver.cc:460] Webserver started at http://127.0.98.1:35321/ using document root <none> and password file <none>
I20260925 13:21:14.337957 19605 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:14.338004 19605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:14.338112 19605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:14.339181 19605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "b902c810807b4cd7b0c4cf085fc1be50"
format_stamp: "Formatted at 2026-09-25 13:21:14 on dist-test-slave-t8d9"
server_key: "36938172adec6fe6f12daf30b726be1e"
server_key_iv: "c0a032b43ccc49926c896b017e0ba50f"
server_key_version: "encryptionkey@0"
I20260925 13:21:14.339512 19605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b902c810807b4cd7b0c4cf085fc1be50"
format_stamp: "Formatted at 2026-09-25 13:21:14 on dist-test-slave-t8d9"
server_key: "36938172adec6fe6f12daf30b726be1e"
server_key_iv: "c0a032b43ccc49926c896b017e0ba50f"
server_key_version: "encryptionkey@0"
I20260925 13:21:14.340802 19605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:21:14.341519 19620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:14.341704 19605 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:14.341789 19605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "b902c810807b4cd7b0c4cf085fc1be50"
format_stamp: "Formatted at 2026-09-25 13:21:14 on dist-test-slave-t8d9"
server_key: "36938172adec6fe6f12daf30b726be1e"
server_key_iv: "c0a032b43ccc49926c896b017e0ba50f"
server_key_version: "encryptionkey@0"
I20260925 13:21:14.341847 19605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:14.368436 19605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:14.370725 19605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:14.370846 19605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:14.371064 19605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:14.371393 19605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:14.371429 19605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:14.371451 19605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:14.371464 19605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:14.377116 19605 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41079
I20260925 13:21:14.377184 19733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41079 every 8 connection(s)
I20260925 13:21:14.377501 19605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 13:21:14 dist-test-slave-t8d9 krb5kdc[19498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:14.386533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 19605
I20260925 13:21:14.386631 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:21:14.386832 392 external_mini_cluster.cc:1428] Setting key 1cb9ab5887c645ccdb07851a9d0c9434
I20260925 13:21:14.387197 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.378255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48801 (local address 127.0.98.62:41613)
0925 13:21:14.378323 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:14.378325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:14.378504 (+ 179us) server_negotiation.cc:407] Connection header received
0925 13:21:14.379479 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:14.379480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:14.379491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:14.379547 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:14.380254 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.380780 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.381881 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.382108 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.382166 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.384497 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:14.384498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:14.384498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:14.384506 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:14.385783 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:14.386165 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:14.386166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:14.386166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:14.386218 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:14.386471 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:14.386472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:14.386472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:14.386579 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:14.386638 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:14.387018 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 13:21:14.387122 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 13:21:14.387399 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.378210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41613 (local address 127.0.98.1:48801)
0925 13:21:14.378382 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:14.378422 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:14.379352 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:14.379610 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:14.379614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:14.379942 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:14.380152 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.380157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.380882 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.380883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.381787 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.381790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.382191 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.382192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.382323 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.382881 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:14.382892 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:14.384392 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:14.385858 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:14.385868 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:14.385871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:14.386103 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:14.386266 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:14.386267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:14.386267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:14.386411 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:14.386640 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:14.386647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:14.386830 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 13:21:14.386981 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 13:21:14.387143 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260925 13:21:14.387913 19734 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41613
I20260925 13:21:14.388017 19734 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:14.388203 19734 heartbeater.cc:511] Master 127.0.98.62:41613 requested a full tablet report, sending...
I20260925 13:21:14.388795 19543 ts_manager.cc:194] Registered new tserver with Master: b902c810807b4cd7b0c4cf085fc1be50 (127.0.98.1:41079)
I20260925 13:21:14.389482 19543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48801
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 13:21:14.413249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41613
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:14.495867 19739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:14.496075 19739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:14.496097 19739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:14.497221 19739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:14.497259 19739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:14.497283 19739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:21:14.498790 19739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:14.498981 19739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:14.499166 19739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:21:14.503813 19739 server_base.cc:1034] running on GCE node
W20260925 13:21:14.503824 19744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:14.503796 19747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:14.503788 19745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:14.504205 19739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:14.504438 19739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:14.505590 19739 hybrid_clock.cc:648] HybridClock initialized: now 1790342474505578 us; error 32 us; skew 500 ppm
Sep 25 13:21:14 dist-test-slave-t8d9 krb5kdc[19498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:14.507838 19739 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 13:21:14.508292 19739 webserver.cc:460] Webserver started at http://127.0.98.2:41677/ using document root <none> and password file <none>
I20260925 13:21:14.508499 19739 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:14.508545 19739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:14.508670 19739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:14.509761 19739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "da66609e55104cac89057fc53d76e770"
format_stamp: "Formatted at 2026-09-25 13:21:14 on dist-test-slave-t8d9"
server_key: "681520efdfeceb3907cecf02dc657d84"
server_key_iv: "0736d2b7ad97180d88b4af77c8fff759"
server_key_version: "encryptionkey@0"
I20260925 13:21:14.510089 19739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "da66609e55104cac89057fc53d76e770"
format_stamp: "Formatted at 2026-09-25 13:21:14 on dist-test-slave-t8d9"
server_key: "681520efdfeceb3907cecf02dc657d84"
server_key_iv: "0736d2b7ad97180d88b4af77c8fff759"
server_key_version: "encryptionkey@0"
I20260925 13:21:14.511307 19739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:21:14.511958 19754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:14.512102 19739 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:14.512181 19739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "da66609e55104cac89057fc53d76e770"
format_stamp: "Formatted at 2026-09-25 13:21:14 on dist-test-slave-t8d9"
server_key: "681520efdfeceb3907cecf02dc657d84"
server_key_iv: "0736d2b7ad97180d88b4af77c8fff759"
server_key_version: "encryptionkey@0"
I20260925 13:21:14.512246 19739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:14.535174 19739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:14.537312 19739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:14.537469 19739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:14.537721 19739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:14.538053 19739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:14.538089 19739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:14.538120 19739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:14.538142 19739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:14.543684 19739 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32979
I20260925 13:21:14.543740 19867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32979 every 8 connection(s)
I20260925 13:21:14.544036 19739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:21:14.548775 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 19739
I20260925 13:21:14.548887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:21:14.549080 392 external_mini_cluster.cc:1428] Setting key 423f0ac5f5c6c1132de4e528f64f57ae
Sep 25 13:21:14 dist-test-slave-t8d9 krb5kdc[19498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:14.553118 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.544750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57459 (local address 127.0.98.62:41613)
0925 13:21:14.544809 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:14.544810 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:14.544998 (+ 188us) server_negotiation.cc:407] Connection header received
0925 13:21:14.546029 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:14.546030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:14.546041 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:14.546081 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:14.546664 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.547179 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.548106 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.548283 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.548327 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.550514 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:14.550516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:14.550516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:14.550524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:14.551814 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:14.552238 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:14.552238 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:14.552238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:14.552283 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:14.552505 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:14.552506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:14.552506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:14.552600 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:14.552643 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:14.552963 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 13:21:14.553052 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:21:14.553344 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.544717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41613 (local address 127.0.98.2:57459)
0925 13:21:14.544874 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:14.544916 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:14.545917 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:14.546141 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:14.546144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:14.546398 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:14.546604 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.546608 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.547240 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.547240 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.548044 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.548046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.548336 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.548336 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.548438 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.548927 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:14.548937 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:14.550432 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:14.551888 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:14.551894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:14.551896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:14.552172 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:14.552331 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:14.552331 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:14.552332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:14.552453 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:14.552648 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:14.552652 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:14.552813 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 13:21:14.552943 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 13:21:14.553112 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260925 13:21:14.553798 19868 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41613
I20260925 13:21:14.553894 19868 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:14.554069 19868 heartbeater.cc:511] Master 127.0.98.62:41613 requested a full tablet report, sending...
I20260925 13:21:14.554446 19543 ts_manager.cc:194] Registered new tserver with Master: da66609e55104cac89057fc53d76e770 (127.0.98.2:32979)
I20260925 13:21:14.554811 19543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57459
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 13:21:14.573513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41613
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:14.681389 19873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:14.681638 19873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:14.681664 19873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:14.682819 19873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:14.682859 19873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:14.682886 19873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:21:14.684389 19873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:14.684621 19873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:14.684834 19873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:14.689695 19879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:14.689695 19881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:14.689754 19878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:14.689728 19873 server_base.cc:1034] running on GCE node
I20260925 13:21:14.690099 19873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:14.690320 19873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:14.691474 19873 hybrid_clock.cc:648] HybridClock initialized: now 1790342474691456 us; error 28 us; skew 500 ppm
Sep 25 13:21:14 dist-test-slave-t8d9 krb5kdc[19498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:14.693917 19873 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 13:21:14.694383 19873 webserver.cc:460] Webserver started at http://127.0.98.3:39863/ using document root <none> and password file <none>
I20260925 13:21:14.694585 19873 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:14.694634 19873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:14.694758 19873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:14.695830 19873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "34c1e5ef413e4378b8ca782ecf2bb56e"
format_stamp: "Formatted at 2026-09-25 13:21:14 on dist-test-slave-t8d9"
server_key: "e1a9759a942dfb91094e14046849f3dc"
server_key_iv: "65b49cc21a0dc461424f12ae1e05f29f"
server_key_version: "encryptionkey@0"
I20260925 13:21:14.696167 19873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "34c1e5ef413e4378b8ca782ecf2bb56e"
format_stamp: "Formatted at 2026-09-25 13:21:14 on dist-test-slave-t8d9"
server_key: "e1a9759a942dfb91094e14046849f3dc"
server_key_iv: "65b49cc21a0dc461424f12ae1e05f29f"
server_key_version: "encryptionkey@0"
I20260925 13:21:14.697499 19873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:21:14.698386 19888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:14.698551 19873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:21:14.698642 19873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "34c1e5ef413e4378b8ca782ecf2bb56e"
format_stamp: "Formatted at 2026-09-25 13:21:14 on dist-test-slave-t8d9"
server_key: "e1a9759a942dfb91094e14046849f3dc"
server_key_iv: "65b49cc21a0dc461424f12ae1e05f29f"
server_key_version: "encryptionkey@0"
I20260925 13:21:14.698698 19873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:14.727748 19873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:14.729933 19873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:14.730046 19873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:14.730226 19873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:14.730482 19873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:14.730513 19873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:14.730533 19873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:14.730549 19873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:14.735908 19873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44033
I20260925 13:21:14.735991 20001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44033 every 8 connection(s)
I20260925 13:21:14.736270 19873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:21:14.739106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 19873
I20260925 13:21:14.739179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:21:14.739323 392 external_mini_cluster.cc:1428] Setting key cb835fb0be07d1bb23643e2e4263d9f6
Sep 25 13:21:14 dist-test-slave-t8d9 krb5kdc[19498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:14.745982 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.737012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57903 (local address 127.0.98.62:41613)
0925 13:21:14.737078 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:14.737080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:14.737253 (+ 173us) server_negotiation.cc:407] Connection header received
0925 13:21:14.738261 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:14.738262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:14.738272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:14.738330 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:14.738990 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.739608 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.740799 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.741000 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.741064 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.743210 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:14.743211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:14.743211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:14.743219 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:14.744578 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:14.745036 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:14.745037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:14.745038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:14.745118 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:14.745333 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:14.745334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:14.745335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:14.745466 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:14.745521 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:14.745811 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 13:21:14.745891 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 13:21:14.746151 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.736951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41613 (local address 127.0.98.3:57903)
0925 13:21:14.737124 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:14.737169 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:14.738155 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:14.738400 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:14.738404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:14.738666 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:14.738904 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.738910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.739674 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.739675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.740705 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.740708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.741081 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.741082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.741190 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.741696 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:14.741704 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:14.743110 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:14.744654 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:14.744660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:14.744661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:14.744893 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:14.745172 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:14.745173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:14.745174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:14.745275 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:14.745521 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:14.745527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:14.745690 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 13:21:14.745780 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 13:21:14.745946 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260925 13:21:14.746557 20002 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41613
I20260925 13:21:14.746646 20002 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:14.746794 20002 heartbeater.cc:511] Master 127.0.98.62:41613 requested a full tablet report, sending...
I20260925 13:21:14.747092 19543 ts_manager.cc:194] Registered new tserver with Master: 34c1e5ef413e4378b8ca782ecf2bb56e (127.0.98.3:44033)
I20260925 13:21:14.747429 19543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57903
I20260925 13:21:14.751214 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:21:14.758006 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.752444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.0.98.62:41613)
0925 13:21:14.752487 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:14.752488 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:14.752603 (+ 115us) server_negotiation.cc:407] Connection header received
0925 13:21:14.752660 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:14.752660 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:14.752668 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:14.752708 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:14.753273 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.753794 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.754505 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.754674 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.754711 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.755807 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:14.755808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:14.755808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:14.755814 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:14.756881 (+ 1067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:14.757207 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:14.757208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:14.757209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:14.757266 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:14.757453 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:14.757454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:14.757454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:14.757671 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:14.757751 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:14.757858 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 13:21:14.757940 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 13:21:14.759423 19543 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:21:14.761160 19543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:21:14.768208 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.763842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.0.98.2:32979)
0925 13:21:14.763995 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:14.764000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:14.764545 (+ 545us) server_negotiation.cc:407] Connection header received
0925 13:21:14.764594 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:14.764597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:14.764613 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:14.764683 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:14.764918 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.765647 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.767123 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.767802 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.767868 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.767928 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:14.768072 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 13:21:14.768116 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260925 13:21:14.768217 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.763555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32979 (local address 127.0.0.1:44072)
0925 13:21:14.764489 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:14.764498 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:14.764542 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:14.764769 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:14.764770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:14.764772 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:14.764829 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.764831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.765733 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.765734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.767042 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.767044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.767871 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.767872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.767956 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.767973 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 13:21:14.768046 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 13:21:14.768113 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":28,"thread_start_us":91,"threads_started":1}
I20260925 13:21:14.768217 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.763551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44033 (local address 127.0.0.1:52652)
0925 13:21:14.764359 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:14.764369 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:14.764413 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:14.764798 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:14.764799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:14.764800 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:14.764834 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.764835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.766063 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.766064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.766949 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.766952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.767844 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.767845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.767956 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.767972 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 13:21:14.768046 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 13:21:14.768113 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260925 13:21:14.768486 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.763976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.0.98.3:44033)
0925 13:21:14.764127 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:14.764130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:14.764591 (+ 461us) server_negotiation.cc:407] Connection header received
0925 13:21:14.764634 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:14.764637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:14.764651 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:14.764713 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:14.764915 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.765977 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.767024 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.767759 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.767832 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.767902 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:14.768342 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 13:21:14.768393 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260925 13:21:14.769093 19936 tablet_service.cc:1467] Processing CreateTablet for tablet 290f29fee7f9405081f3edc410a3572e (DEFAULT_TABLE table=test-table [id=cd0ed597a72844a3ba437be2cc2f4323]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:14.769112 19802 tablet_service.cc:1467] Processing CreateTablet for tablet 290f29fee7f9405081f3edc410a3572e (DEFAULT_TABLE table=test-table [id=cd0ed597a72844a3ba437be2cc2f4323]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:14.769433 19936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 290f29fee7f9405081f3edc410a3572e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:14.769462 19802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 290f29fee7f9405081f3edc410a3572e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:14.769779 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.763550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41079 (local address 127.0.0.1:55888)
0925 13:21:14.764125 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:14.764159 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:14.764229 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:14.764463 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:14.764467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:14.764470 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:14.764551 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.764553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.765943 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.765944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.767042 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.767044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.769507 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.769508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.769607 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.769612 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 13:21:14.769660 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 13:21:14.769708 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":29,"thread_start_us":100,"threads_started":1}
I20260925 13:21:14.770016 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.763811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.0.98.1:41079)
0925 13:21:14.763980 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:14.763985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:14.764233 (+ 248us) server_negotiation.cc:407] Connection header received
0925 13:21:14.764319 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:14.764322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:14.764340 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:14.764424 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:14.764641 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.765820 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.767968 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.769332 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.769404 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.769467 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:14.769883 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 13:21:14.769943 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260925 13:21:14.772132 20019 tablet_bootstrap.cc:492] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e: Bootstrap starting.
I20260925 13:21:14.772272 20020 tablet_bootstrap.cc:492] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770: Bootstrap starting.
I20260925 13:21:14.772871 20020 tablet_bootstrap.cc:654] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:14.772816 19668 tablet_service.cc:1467] Processing CreateTablet for tablet 290f29fee7f9405081f3edc410a3572e (DEFAULT_TABLE table=test-table [id=cd0ed597a72844a3ba437be2cc2f4323]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:14.773092 19668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 290f29fee7f9405081f3edc410a3572e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:14.773334 20020 log.cc:824] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:14.773973 20019 tablet_bootstrap.cc:654] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:14.774116 20020 tablet_bootstrap.cc:492] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770: No bootstrap required, opened a new log
I20260925 13:21:14.774188 20020 ts_tablet_manager.cc:1397] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:21:14.774557 20019 log.cc:824] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:14.775321 20019 tablet_bootstrap.cc:492] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e: No bootstrap required, opened a new log
I20260925 13:21:14.775393 20019 ts_tablet_manager.cc:1397] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 13:21:14.775954 20023 tablet_bootstrap.cc:492] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50: Bootstrap starting.
I20260925 13:21:14.776093 20020 raft_consensus.cc:348] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902c810807b4cd7b0c4cf085fc1be50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41079 } } peers { permanent_uuid: "da66609e55104cac89057fc53d76e770" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32979 } } peers { permanent_uuid: "34c1e5ef413e4378b8ca782ecf2bb56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } }
I20260925 13:21:14.776306 20020 raft_consensus.cc:374] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:14.776353 20020 raft_consensus.cc:729] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da66609e55104cac89057fc53d76e770, State: Initialized, Role: FOLLOWER
I20260925 13:21:14.776476 20020 consensus_queue.cc:260] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902c810807b4cd7b0c4cf085fc1be50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41079 } } peers { permanent_uuid: "da66609e55104cac89057fc53d76e770" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32979 } } peers { permanent_uuid: "34c1e5ef413e4378b8ca782ecf2bb56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } }
I20260925 13:21:14.776665 20023 tablet_bootstrap.cc:654] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:14.776763 20020 ts_tablet_manager.cc:1428] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 13:21:14.776906 19868 heartbeater.cc:503] Master 127.0.98.62:41613 was elected leader, sending a full tablet report...
I20260925 13:21:14.777223 20023 log.cc:824] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:14.777289 20019 raft_consensus.cc:348] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902c810807b4cd7b0c4cf085fc1be50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41079 } } peers { permanent_uuid: "da66609e55104cac89057fc53d76e770" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32979 } } peers { permanent_uuid: "34c1e5ef413e4378b8ca782ecf2bb56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } }
I20260925 13:21:14.777474 20019 raft_consensus.cc:374] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:14.777516 20019 raft_consensus.cc:729] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c1e5ef413e4378b8ca782ecf2bb56e, State: Initialized, Role: FOLLOWER
I20260925 13:21:14.777657 20019 consensus_queue.cc:260] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902c810807b4cd7b0c4cf085fc1be50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41079 } } peers { permanent_uuid: "da66609e55104cac89057fc53d76e770" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32979 } } peers { permanent_uuid: "34c1e5ef413e4378b8ca782ecf2bb56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } }
I20260925 13:21:14.777933 20019 ts_tablet_manager.cc:1428] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 13:21:14.777946 20002 heartbeater.cc:503] Master 127.0.98.62:41613 was elected leader, sending a full tablet report...
I20260925 13:21:14.777993 20023 tablet_bootstrap.cc:492] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50: No bootstrap required, opened a new log
I20260925 13:21:14.778048 20023 ts_tablet_manager.cc:1397] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:21:14.779563 20023 raft_consensus.cc:348] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902c810807b4cd7b0c4cf085fc1be50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41079 } } peers { permanent_uuid: "da66609e55104cac89057fc53d76e770" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32979 } } peers { permanent_uuid: "34c1e5ef413e4378b8ca782ecf2bb56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } }
I20260925 13:21:14.779695 20023 raft_consensus.cc:374] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:14.779716 20023 raft_consensus.cc:729] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b902c810807b4cd7b0c4cf085fc1be50, State: Initialized, Role: FOLLOWER
I20260925 13:21:14.779824 20023 consensus_queue.cc:260] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902c810807b4cd7b0c4cf085fc1be50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41079 } } peers { permanent_uuid: "da66609e55104cac89057fc53d76e770" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32979 } } peers { permanent_uuid: "34c1e5ef413e4378b8ca782ecf2bb56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } }
I20260925 13:21:14.780072 20023 ts_tablet_manager.cc:1428] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 13:21:14.780092 19734 heartbeater.cc:503] Master 127.0.98.62:41613 was elected leader, sending a full tablet report...
I20260925 13:21:14.791138 20027 raft_consensus.cc:482] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:21:14.791256 20027 raft_consensus.cc:504] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902c810807b4cd7b0c4cf085fc1be50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41079 } } peers { permanent_uuid: "da66609e55104cac89057fc53d76e770" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32979 } } peers { permanent_uuid: "34c1e5ef413e4378b8ca782ecf2bb56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } }
I20260925 13:21:14.791630 20027 leader_election.cc:288] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da66609e55104cac89057fc53d76e770 (127.0.98.2:32979), 34c1e5ef413e4378b8ca782ecf2bb56e (127.0.98.3:44033)
I20260925 13:21:14.794507 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.791915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58697 (local address 127.0.98.2:32979)
0925 13:21:14.791954 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:14.791956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:14.792165 (+ 209us) server_negotiation.cc:407] Connection header received
0925 13:21:14.792203 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:14.792204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:14.792219 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:14.792272 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:14.792406 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.792883 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.793825 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.794203 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.794265 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.794276 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:14.794396 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 13:21:14.794438 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:21:14.794507 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.791814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44033 (local address 127.0.98.1:41799)
0925 13:21:14.791874 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:14.791884 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:14.791929 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:14.792118 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:14.792119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:14.792123 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:14.792205 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.792207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.792876 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.792877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.793720 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.793723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.794253 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.794254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.794326 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.794329 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:14.794374 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 13:21:14.794428 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 13:21:14.794623 19869 tablet.cc:2369] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:21:14.794674 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.791848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32979 (local address 127.0.98.1:58697)
0925 13:21:14.792035 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:14.792044 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:14.792112 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:14.792279 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:14.792279 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:14.792281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:14.792352 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.792353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.792972 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.792973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.793746 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:14.793748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.794273 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.794274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:14.794333 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.794336 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:14.794382 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 13:21:14.794478 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":111,"threads_started":1}
I20260925 13:21:14.794998 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.791872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41799 (local address 127.0.98.3:44033)
0925 13:21:14.791935 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:14.791937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:14.791958 (+ 21us) server_negotiation.cc:407] Connection header received
0925 13:21:14.791987 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:14.791988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:14.791997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:14.792090 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:14.792281 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.792807 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.793796 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.794182 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.794234 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.794245 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:14.794391 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 13:21:14.794924 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 13:21:14.794979 19822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290f29fee7f9405081f3edc410a3572e" candidate_uuid: "b902c810807b4cd7b0c4cf085fc1be50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da66609e55104cac89057fc53d76e770" is_pre_election: true
I20260925 13:21:14.795126 19822 raft_consensus.cc:2454] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b902c810807b4cd7b0c4cf085fc1be50 in term 0.
I20260925 13:21:14.795354 19621 leader_election.cc:302] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b902c810807b4cd7b0c4cf085fc1be50, da66609e55104cac89057fc53d76e770; no voters:
I20260925 13:21:14.795310 19956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290f29fee7f9405081f3edc410a3572e" candidate_uuid: "b902c810807b4cd7b0c4cf085fc1be50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34c1e5ef413e4378b8ca782ecf2bb56e" is_pre_election: true
I20260925 13:21:14.795470 19956 raft_consensus.cc:2454] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b902c810807b4cd7b0c4cf085fc1be50 in term 0.
I20260925 13:21:14.795472 20027 raft_consensus.cc:2781] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:21:14.795522 20027 raft_consensus.cc:482] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:21:14.795542 20027 raft_consensus.cc:3037] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:14.796208 20027 raft_consensus.cc:504] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902c810807b4cd7b0c4cf085fc1be50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41079 } } peers { permanent_uuid: "da66609e55104cac89057fc53d76e770" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32979 } } peers { permanent_uuid: "34c1e5ef413e4378b8ca782ecf2bb56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } }
I20260925 13:21:14.796336 20027 leader_election.cc:288] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [CANDIDATE]: Term 1 election: Requested vote from peers da66609e55104cac89057fc53d76e770 (127.0.98.2:32979), 34c1e5ef413e4378b8ca782ecf2bb56e (127.0.98.3:44033)
I20260925 13:21:14.796515 19822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290f29fee7f9405081f3edc410a3572e" candidate_uuid: "b902c810807b4cd7b0c4cf085fc1be50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da66609e55104cac89057fc53d76e770"
I20260925 13:21:14.796509 19956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290f29fee7f9405081f3edc410a3572e" candidate_uuid: "b902c810807b4cd7b0c4cf085fc1be50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34c1e5ef413e4378b8ca782ecf2bb56e"
I20260925 13:21:14.796583 19956 raft_consensus.cc:3037] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:14.796586 19822 raft_consensus.cc:3037] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:14.797274 19956 raft_consensus.cc:2454] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b902c810807b4cd7b0c4cf085fc1be50 in term 1.
I20260925 13:21:14.797279 19822 raft_consensus.cc:2454] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b902c810807b4cd7b0c4cf085fc1be50 in term 1.
I20260925 13:21:14.797461 19623 leader_election.cc:302] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34c1e5ef413e4378b8ca782ecf2bb56e, b902c810807b4cd7b0c4cf085fc1be50; no voters:
I20260925 13:21:14.797569 20027 raft_consensus.cc:2781] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:14.797715 20027 raft_consensus.cc:686] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [term 1 LEADER]: Becoming Leader. State: Replica: b902c810807b4cd7b0c4cf085fc1be50, State: Running, Role: LEADER
I20260925 13:21:14.797793 20027 consensus_queue.cc:237] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902c810807b4cd7b0c4cf085fc1be50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41079 } } peers { permanent_uuid: "da66609e55104cac89057fc53d76e770" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32979 } } peers { permanent_uuid: "34c1e5ef413e4378b8ca782ecf2bb56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } }
I20260925 13:21:14.798619 19543 catalog_manager.cc:5495] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 reported cstate change: term changed from 0 to 1, leader changed from <none> to b902c810807b4cd7b0c4cf085fc1be50 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b902c810807b4cd7b0c4cf085fc1be50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b902c810807b4cd7b0c4cf085fc1be50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da66609e55104cac89057fc53d76e770" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34c1e5ef413e4378b8ca782ecf2bb56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } health_report { overall_health: UNKNOWN } } }
I20260925 13:21:14.806813 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.804654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.0.98.1:41079)
0925 13:21:14.804705 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:14.804707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:14.804715 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:21:14.804771 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:14.804772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:14.804790 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:14.804848 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:14.804957 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.805409 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.806138 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.806330 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.806378 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.806508 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:14.806609 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:14.806664 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:14.806701 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 13:21:14.806719 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:21:14.808558 19956 raft_consensus.cc:1261] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e [term 1 FOLLOWER]: Refusing update from remote peer b902c810807b4cd7b0c4cf085fc1be50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:14.808562 19822 raft_consensus.cc:1261] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770 [term 1 FOLLOWER]: Refusing update from remote peer b902c810807b4cd7b0c4cf085fc1be50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:14.808892 20029 consensus_queue.cc:1035] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da66609e55104cac89057fc53d76e770" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:21:14.809023 20027 consensus_queue.cc:1035] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34c1e5ef413e4378b8ca782ecf2bb56e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:21:14.812294 20033 mvcc.cc:205] Tried to move back new op lower bound from 7333242776813379584 to 7333242776772530176. Current Snapshot: MvccSnapshot[applied={T|T < 7333242776813379584 or (T in {7333242776813379584})}]
I20260925 13:21:14.813210 19543 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:21:14.813299 19543 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:21:14.814059 19543 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3434a146b3d64b6bba937ef15221620b: Sending DeleteTablet for 3 replicas of tablet 290f29fee7f9405081f3edc410a3572e
I20260925 13:21:14.814391 19802 tablet_service.cc:1514] Processing DeleteTablet for tablet 290f29fee7f9405081f3edc410a3572e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44072
I20260925 13:21:14.814427 19936 tablet_service.cc:1514] Processing DeleteTablet for tablet 290f29fee7f9405081f3edc410a3572e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52652
I20260925 13:21:14.814580 19668 tablet_service.cc:1514] Processing DeleteTablet for tablet 290f29fee7f9405081f3edc410a3572e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55888
I20260925 13:21:14.814648 20041 tablet_replica.cc:330] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770: stopping tablet replica
I20260925 13:21:14.814739 20041 raft_consensus.cc:2229] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:21:14.814857 20042 tablet_replica.cc:330] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e: stopping tablet replica
I20260925 13:21:14.814961 20043 tablet_replica.cc:330] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50: stopping tablet replica
I20260925 13:21:14.815037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 19605
I20260925 13:21:14.815044 20043 raft_consensus.cc:2229] T 290f29fee7f9405081f3edc410a3572e P b902c810807b4cd7b0c4cf085fc1be50 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:21:14.816246 20041 raft_consensus.cc:2258] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:14.816641 20041 ts_tablet_manager.cc:1905] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:21:14.816792 20042 raft_consensus.cc:2229] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:21:14.816990 20042 raft_consensus.cc:2258] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:14.817502 20042 ts_tablet_manager.cc:1905] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:21:14.818044 20041 ts_tablet_manager.cc:1918] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:21:14.818101 20041 log.cc:1196] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-1/wal/wals/290f29fee7f9405081f3edc410a3572e
I20260925 13:21:14.818279 20041 ts_tablet_manager.cc:1939] T 290f29fee7f9405081f3edc410a3572e P da66609e55104cac89057fc53d76e770: Deleting consensus metadata
I20260925 13:21:14.818703 19527 catalog_manager.cc:4841] TS da66609e55104cac89057fc53d76e770 (127.0.98.2:32979): tablet 290f29fee7f9405081f3edc410a3572e (table test-table [id=cd0ed597a72844a3ba437be2cc2f4323]) successfully deleted
I20260925 13:21:14.819299 20042 ts_tablet_manager.cc:1918] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:21:14.819372 20042 log.cc:1196] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/290f29fee7f9405081f3edc410a3572e
I20260925 13:21:14.819613 20042 ts_tablet_manager.cc:1939] T 290f29fee7f9405081f3edc410a3572e P 34c1e5ef413e4378b8ca782ecf2bb56e: Deleting consensus metadata
I20260925 13:21:14.819998 19529 catalog_manager.cc:4841] TS 34c1e5ef413e4378b8ca782ecf2bb56e (127.0.98.3:44033): tablet 290f29fee7f9405081f3edc410a3572e (table test-table [id=cd0ed597a72844a3ba437be2cc2f4323]) successfully deleted
W20260925 13:21:14.820783 19528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41079 (error 108)
I20260925 13:21:14.821271 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.821154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41079 (local address 127.0.0.1:55918)
0925 13:21:14.821198 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:14.821234 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 13:21:14.821309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 19739
W20260925 13:21:14.821408 19528 catalog_manager.cc:4584] TS b902c810807b4cd7b0c4cf085fc1be50 (127.0.98.1:41079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 290f29fee7f9405081f3edc410a3572e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41079: connect: Connection refused (error 111)
I20260925 13:21:14.826192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 19873
I20260925 13:21:14.829954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 19511
2026-09-25T13:21:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1871 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T13:21:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:14Z Disabled control of system clock
I20260925 13:21:14.855415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45943
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:21:14.934559 20048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:14.934789 20048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:14.934809 20048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:14.935926 20048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:14.935981 20048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:14.935994 20048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:14.936008 20048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:14.936024 20048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:14.937322 20048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45943
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:14.937563 20048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:14.937829 20048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:14.942400 20053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:14.942400 20056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:14.942433 20054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:14.942430 20048 server_base.cc:1034] running on GCE node
I20260925 13:21:14.942870 20048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:14.943173 20048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:14.944350 20048 hybrid_clock.cc:648] HybridClock initialized: now 1790342474944321 us; error 47 us; skew 500 ppm
I20260925 13:21:14.945708 20048 webserver.cc:460] Webserver started at http://127.0.98.62:34345/ using document root <none> and password file <none>
I20260925 13:21:14.945966 20048 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:14.946018 20048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:14.946142 20048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:14.947209 20048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "3afe58d10dde43ffa2d756179acbda50"
format_stamp: "Formatted at 2026-09-25 13:21:14 on dist-test-slave-t8d9"
server_key: "4492720adb97432c8112e1398797b7f3"
server_key_iv: "a76edcc686dc51f35d9d254a075710f9"
server_key_version: "encryptionkey@0"
I20260925 13:21:14.947540 20048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "3afe58d10dde43ffa2d756179acbda50"
format_stamp: "Formatted at 2026-09-25 13:21:14 on dist-test-slave-t8d9"
server_key: "4492720adb97432c8112e1398797b7f3"
server_key_iv: "a76edcc686dc51f35d9d254a075710f9"
server_key_version: "encryptionkey@0"
I20260925 13:21:14.948748 20048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:21:14.949446 20062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:14.949635 20048 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:21:14.949717 20048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "3afe58d10dde43ffa2d756179acbda50"
format_stamp: "Formatted at 2026-09-25 13:21:14 on dist-test-slave-t8d9"
server_key: "4492720adb97432c8112e1398797b7f3"
server_key_iv: "a76edcc686dc51f35d9d254a075710f9"
server_key_version: "encryptionkey@0"
I20260925 13:21:14.949785 20048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:14.966662 20048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:14.967017 20048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:14.967154 20048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:14.971006 20048 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45943
I20260925 13:21:14.971041 20114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45943 every 8 connection(s)
I20260925 13:21:14.971369 20048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:14.972276 20115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:14.974545 20115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50: Bootstrap starting.
I20260925 13:21:14.975124 20115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:14.975404 20115 log.cc:824] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:14.976145 20115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50: No bootstrap required, opened a new log
I20260925 13:21:14.977804 20115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afe58d10dde43ffa2d756179acbda50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45943 } }
I20260925 13:21:14.977948 20115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:14.977975 20115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3afe58d10dde43ffa2d756179acbda50, State: Initialized, Role: FOLLOWER
I20260925 13:21:14.978133 20115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afe58d10dde43ffa2d756179acbda50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45943 } }
I20260925 13:21:14.978199 20115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:14.978221 20115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:14.978267 20115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:14.978801 20115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afe58d10dde43ffa2d756179acbda50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45943 } }
I20260925 13:21:14.978886 20115 leader_election.cc:302] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3afe58d10dde43ffa2d756179acbda50; no voters:
I20260925 13:21:14.979024 20115 leader_election.cc:288] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:14.979074 20118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:14.979233 20118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [term 1 LEADER]: Becoming Leader. State: Replica: 3afe58d10dde43ffa2d756179acbda50, State: Running, Role: LEADER
I20260925 13:21:14.979275 20115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:14.979332 20118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afe58d10dde43ffa2d756179acbda50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45943 } }
I20260925 13:21:14.979679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 20048
I20260925 13:21:14.979794 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:14.979820 20119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3afe58d10dde43ffa2d756179acbda50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afe58d10dde43ffa2d756179acbda50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45943 } } }
I20260925 13:21:14.979893 20119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:14.979983 392 external_mini_cluster.cc:1428] Setting key 6eb85820f1bd6906ab38cb13adbd9dd9
I20260925 13:21:14.980062 20120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3afe58d10dde43ffa2d756179acbda50. Latest consensus state: current_term: 1 leader_uuid: "3afe58d10dde43ffa2d756179acbda50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afe58d10dde43ffa2d756179acbda50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45943 } } }
I20260925 13:21:14.980118 20120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:14.980219 20127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:14.980697 20127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:14.983073 20127 catalog_manager.cc:1347] Generated new cluster ID: 46aa21f423904ce3a7c31a0a0efe43f3
I20260925 13:21:14.983124 20127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:14.985780 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:14.980627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.0.98.62:45943)
0925 13:21:14.980864 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:14.980874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:14.981027 (+ 153us) server_negotiation.cc:407] Connection header received
0925 13:21:14.981250 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:14.981258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:14.981335 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:14.981475 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:14.981751 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.982635 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.983663 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:14.984295 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:14.984429 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:14.985044 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:14.985048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:14.985048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:14.985252 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:14.985295 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:14.985311 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:14.985379 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:14.985441 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 13:21:14.985520 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
I20260925 13:21:15.012133 20127 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:15.012941 20127 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:15.016779 20127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50: Generated new TSK 0
I20260925 13:21:15.017026 20127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:15.023003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45943
--builtin_ntp_servers=127.0.98.20:38887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:21:15.105077 20139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:15.105309 20139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:15.105340 20139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:15.106648 20139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:15.106701 20139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:15.106739 20139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 13:21:15.108183 20139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:15.108438 20139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:15.108736 20139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:15.113528 20144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:15.113554 20145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:15.113602 20147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:15.113941 20139 server_base.cc:1034] running on GCE node
I20260925 13:21:15.114159 20139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:15.114379 20139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:15.115550 20139 hybrid_clock.cc:648] HybridClock initialized: now 1790342475115535 us; error 24 us; skew 500 ppm
I20260925 13:21:15.116830 20139 webserver.cc:460] Webserver started at http://127.0.98.1:40177/ using document root <none> and password file <none>
I20260925 13:21:15.117093 20139 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:15.117146 20139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:15.117272 20139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:15.118366 20139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/data/instance:
uuid: "71eb88bfa015496ea798a859da25e218"
format_stamp: "Formatted at 2026-09-25 13:21:15 on dist-test-slave-t8d9"
server_key: "c8e088663bdf853953f6b549491fd9b1"
server_key_iv: "767841fc92affad55ed0a3913395cf2e"
server_key_version: "encryptionkey@0"
I20260925 13:21:15.118686 20139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/wal/instance:
uuid: "71eb88bfa015496ea798a859da25e218"
format_stamp: "Formatted at 2026-09-25 13:21:15 on dist-test-slave-t8d9"
server_key: "c8e088663bdf853953f6b549491fd9b1"
server_key_iv: "767841fc92affad55ed0a3913395cf2e"
server_key_version: "encryptionkey@0"
I20260925 13:21:15.119890 20139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 13:21:15.120569 20153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:15.120750 20139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:15.120829 20139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/wal
uuid: "71eb88bfa015496ea798a859da25e218"
format_stamp: "Formatted at 2026-09-25 13:21:15 on dist-test-slave-t8d9"
server_key: "c8e088663bdf853953f6b549491fd9b1"
server_key_iv: "767841fc92affad55ed0a3913395cf2e"
server_key_version: "encryptionkey@0"
I20260925 13:21:15.120886 20139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:15.138092 20139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:15.138412 20139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:15.138545 20139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:15.138772 20139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:15.139132 20139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:15.139166 20139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:15.139195 20139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:15.139211 20139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:15.144567 20139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45847
I20260925 13:21:15.144630 20266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45847 every 8 connection(s)
I20260925 13:21:15.144996 20139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/data/info.pb
I20260925 13:21:15.148150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 20139
I20260925 13:21:15.148237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-0/wal/instance
I20260925 13:21:15.148423 392 external_mini_cluster.cc:1428] Setting key e2caa24c11f5af1379dc9f636335f39b
I20260925 13:21:15.149284 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45943
--builtin_ntp_servers=127.0.98.20:38887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 13:21:15.150393 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.145788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56183 (local address 127.0.98.62:45943)
0925 13:21:15.145854 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:15.145856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:15.146108 (+ 252us) server_negotiation.cc:407] Connection header received
0925 13:21:15.146617 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:15.146618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:15.146633 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:15.146683 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:15.147346 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.147847 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.148990 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.149178 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.149227 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.150141 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:15.150143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:15.150143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:15.150148 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:15.150163 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:15.150167 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:15.150206 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:15.150288 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 13:21:15.150317 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:21:15.150609 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.145787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45943 (local address 127.0.98.1:56183)
0925 13:21:15.145991 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:15.146030 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:15.146517 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:15.146757 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:15.146761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:15.146771 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:15.147257 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.147262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.147938 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.147939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.148857 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.148860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.149243 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.149244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.149404 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.150010 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:21:15.150011 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:15.150041 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:21:15.150065 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:15.150223 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:15.150227 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:15.150230 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:15.150284 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 13:21:15.150385 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260925 13:21:15.151212 20267 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45943
I20260925 13:21:15.151296 20267 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:15.151471 20267 heartbeater.cc:511] Master 127.0.98.62:45943 requested a full tablet report, sending...
I20260925 13:21:15.151971 20079 ts_manager.cc:194] Registered new tserver with Master: 71eb88bfa015496ea798a859da25e218 (127.0.98.1:45847)
I20260925 13:21:15.152680 20079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56183
W20260925 13:21:15.230878 20270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:15.231102 20270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:15.231123 20270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:15.232329 20270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:15.232378 20270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:15.232403 20270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 13:21:15.233851 20270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:15.234097 20270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:15.234405 20270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:15.239399 20278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:15.239432 20276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:15.239396 20275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:15.239624 20270 server_base.cc:1034] running on GCE node
I20260925 13:21:15.239832 20270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:15.240056 20270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:15.241230 20270 hybrid_clock.cc:648] HybridClock initialized: now 1790342475241213 us; error 30 us; skew 500 ppm
I20260925 13:21:15.242594 20270 webserver.cc:460] Webserver started at http://127.0.98.2:34897/ using document root <none> and password file <none>
I20260925 13:21:15.242826 20270 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:15.242874 20270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:15.243000 20270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:15.244068 20270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/data/instance:
uuid: "84056f14ca49411ea762811d439f9081"
format_stamp: "Formatted at 2026-09-25 13:21:15 on dist-test-slave-t8d9"
server_key: "4930bc4473d6a9b9263aa1e2324d02e3"
server_key_iv: "ee135b8d8cad6e6264cfaaec17de9d5c"
server_key_version: "encryptionkey@0"
I20260925 13:21:15.244395 20270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84056f14ca49411ea762811d439f9081"
format_stamp: "Formatted at 2026-09-25 13:21:15 on dist-test-slave-t8d9"
server_key: "4930bc4473d6a9b9263aa1e2324d02e3"
server_key_iv: "ee135b8d8cad6e6264cfaaec17de9d5c"
server_key_version: "encryptionkey@0"
I20260925 13:21:15.245690 20270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:21:15.246505 20284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:15.246655 20270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:15.246738 20270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/wal
uuid: "84056f14ca49411ea762811d439f9081"
format_stamp: "Formatted at 2026-09-25 13:21:15 on dist-test-slave-t8d9"
server_key: "4930bc4473d6a9b9263aa1e2324d02e3"
server_key_iv: "ee135b8d8cad6e6264cfaaec17de9d5c"
server_key_version: "encryptionkey@0"
I20260925 13:21:15.246840 20270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:15.258317 20270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:15.258630 20270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:15.258754 20270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:15.258981 20270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:15.259294 20270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:15.259330 20270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:15.259358 20270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:15.259380 20270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:15.264779 20270 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45179
I20260925 13:21:15.264827 20397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45179 every 8 connection(s)
I20260925 13:21:15.265143 20270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/data/info.pb
I20260925 13:21:15.270177 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.265856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48007 (local address 127.0.98.62:45943)
0925 13:21:15.265941 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:15.265943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:15.266139 (+ 196us) server_negotiation.cc:407] Connection header received
0925 13:21:15.266635 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:15.266636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:15.266648 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:15.266698 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:15.267365 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.267910 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.268895 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.269090 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.269152 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.269946 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:15.269947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:15.269947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:15.269952 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:15.269968 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:15.269972 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:15.270015 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:15.270089 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 13:21:15.270115 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 13:21:15.270359 20400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.265844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45943 (local address 127.0.98.2:48007)
0925 13:21:15.266023 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:15.266060 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:15.266546 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:15.266755 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:15.266758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:15.266766 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:15.267251 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.267255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.268012 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.268013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.268811 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.268813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.269161 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.269162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.269271 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.269814 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:21:15.269814 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:15.269844 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:21:15.269868 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:15.270021 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:15.270024 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:15.270027 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:15.270086 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 13:21:15.270168 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260925 13:21:15.270812 20398 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45943
I20260925 13:21:15.270898 20398 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:15.271065 20398 heartbeater.cc:511] Master 127.0.98.62:45943 requested a full tablet report, sending...
I20260925 13:21:15.271445 20079 ts_manager.cc:194] Registered new tserver with Master: 84056f14ca49411ea762811d439f9081 (127.0.98.2:45179)
I20260925 13:21:15.271806 20079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48007
I20260925 13:21:15.274231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 20270
I20260925 13:21:15.274328 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-1/wal/instance
I20260925 13:21:15.274504 392 external_mini_cluster.cc:1428] Setting key 631a966e59fc83930c108bc8186728c9
I20260925 13:21:15.275445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45943
--builtin_ntp_servers=127.0.98.20:38887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 13:21:15.356550 20401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:15.356775 20401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:15.356806 20401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:15.358207 20401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:15.358273 20401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:15.358314 20401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 13:21:15.359814 20401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:15.360060 20401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:15.360351 20401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:15.365293 20409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:15.365448 20407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:15.365432 20401 server_base.cc:1034] running on GCE node
W20260925 13:21:15.365373 20406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:15.365746 20401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:15.366014 20401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:15.367154 20401 hybrid_clock.cc:648] HybridClock initialized: now 1790342475367134 us; error 37 us; skew 500 ppm
I20260925 13:21:15.368348 20401 webserver.cc:460] Webserver started at http://127.0.98.3:41767/ using document root <none> and password file <none>
I20260925 13:21:15.368577 20401 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:15.368626 20401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:15.368757 20401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:15.369902 20401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/data/instance:
uuid: "fde8289cf0784c8e86163f2270068b8e"
format_stamp: "Formatted at 2026-09-25 13:21:15 on dist-test-slave-t8d9"
server_key: "0fb550167b932a97c6e768c573e0129d"
server_key_iv: "6fe3657fa041393e1fc2a78af579521c"
server_key_version: "encryptionkey@0"
I20260925 13:21:15.370249 20401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fde8289cf0784c8e86163f2270068b8e"
format_stamp: "Formatted at 2026-09-25 13:21:15 on dist-test-slave-t8d9"
server_key: "0fb550167b932a97c6e768c573e0129d"
server_key_iv: "6fe3657fa041393e1fc2a78af579521c"
server_key_version: "encryptionkey@0"
I20260925 13:21:15.371479 20401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 13:21:15.372195 20415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:15.372356 20401 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:21:15.372438 20401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/wal
uuid: "fde8289cf0784c8e86163f2270068b8e"
format_stamp: "Formatted at 2026-09-25 13:21:15 on dist-test-slave-t8d9"
server_key: "0fb550167b932a97c6e768c573e0129d"
server_key_iv: "6fe3657fa041393e1fc2a78af579521c"
server_key_version: "encryptionkey@0"
I20260925 13:21:15.372500 20401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:15.394124 20401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:15.394456 20401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:15.394593 20401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:15.394835 20401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 13:21:15.395185 20401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 13:21:15.395220 20401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:15.395249 20401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 13:21:15.395265 20401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:15.400657 20401 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44361
I20260925 13:21:15.400718 20528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44361 every 8 connection(s)
I20260925 13:21:15.401054 20401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/data/info.pb
I20260925 13:21:15.406229 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.401802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49507 (local address 127.0.98.62:45943)
0925 13:21:15.401858 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:15.401860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:15.402090 (+ 230us) server_negotiation.cc:407] Connection header received
0925 13:21:15.402566 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:15.402567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:15.402576 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:15.402638 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:15.403290 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.403793 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.404747 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.404931 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.404977 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.405952 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:15.405954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:15.405954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:15.405959 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:15.405974 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:15.405978 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:15.406014 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:15.406134 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 13:21:15.406165 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 13:21:15.406453 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.401785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45943 (local address 127.0.98.3:49507)
0925 13:21:15.401969 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:15.402012 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:15.402487 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:15.402696 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:15.402699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:15.402709 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:15.403218 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.403222 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.403863 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.403863 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.404679 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.404681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.405012 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.405012 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.405127 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.405734 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 13:21:15.405735 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:15.405769 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 13:21:15.405832 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:15.406049 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:15.406056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:15.406059 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:15.406133 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 13:21:15.406226 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260925 13:21:15.406944 20529 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45943
I20260925 13:21:15.407040 20529 heartbeater.cc:464] Registering TS with master...
I20260925 13:21:15.407194 20529 heartbeater.cc:511] Master 127.0.98.62:45943 requested a full tablet report, sending...
I20260925 13:21:15.407559 20079 ts_manager.cc:194] Registered new tserver with Master: fde8289cf0784c8e86163f2270068b8e (127.0.98.3:44361)
I20260925 13:21:15.407891 20079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49507
I20260925 13:21:15.410534 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 20401
I20260925 13:21:15.410630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/wal/instance
I20260925 13:21:15.410799 392 external_mini_cluster.cc:1428] Setting key 259f7a3c51b900bdeccd42ef59ca38b7
I20260925 13:21:15.412014 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 13:21:15.416103 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.413410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.0.98.62:45943)
0925 13:21:15.413467 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:15.413468 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:15.413513 (+ 45us) server_negotiation.cc:407] Connection header received
0925 13:21:15.413604 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:15.413605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:15.413615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:15.413667 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:15.413809 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.414287 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.415021 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.415213 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.415273 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.415911 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:15.415913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:15.415913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:15.415917 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:15.415931 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:15.415935 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:15.415975 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:15.416015 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 13:21:15.416042 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:21:15.417610 20079 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 13:21:15.419215 20079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 13:21:15.425284 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.421651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.0.98.3:44361)
0925 13:21:15.421834 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:15.421839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:15.421848 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:21:15.421887 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:15.421889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:15.421903 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:15.421981 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:15.422201 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.423265 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.424252 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.424893 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.424965 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.425066 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:15.425149 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 13:21:15.425188 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
I20260925 13:21:15.425269 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.421503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44361 (local address 127.0.0.1:35372)
0925 13:21:15.421678 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:15.421687 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:15.421731 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:15.422105 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:15.422105 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:15.422107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:15.422131 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.422132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.423343 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.423344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.424168 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.424170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.424978 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.424979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.425058 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.425073 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 13:21:15.425137 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 13:21:15.425191 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
I20260925 13:21:15.425516 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.421269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45179 (local address 127.0.0.1:41764)
0925 13:21:15.421399 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:15.421445 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:15.421511 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:15.421979 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:15.421980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:15.421982 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:15.422011 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.422012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.422965 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.422966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.424189 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.424192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.425267 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.425268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.425332 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.425336 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:21:15.425365 (+ 29us) client_negotiation.cc:240] Negotiation successful
0925 13:21:15.425438 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":82,"threads_started":1}
I20260925 13:21:15.425844 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.421268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45847 (local address 127.0.0.1:59168)
0925 13:21:15.421644 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:15.421654 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:15.421702 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:15.421879 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:15.421881 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:15.421883 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:15.421980 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.421982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.423301 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.423302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.424354 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.424356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.425513 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.425514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.425681 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.425685 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:21:15.425736 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 13:21:15.425776 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
I20260925 13:21:15.425885 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.421372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.0.98.1:45847)
0925 13:21:15.421721 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:15.421726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:15.421737 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:21:15.421776 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:15.421778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:15.421793 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:15.421863 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:15.422065 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.423212 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.424417 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.425169 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.425238 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.425482 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:15.425757 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 13:21:15.425804 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":93,"threads_started":1}
I20260925 13:21:15.426021 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.421477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.0.98.2:45179)
0925 13:21:15.421849 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:15.421853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:15.421861 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:21:15.421893 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:15.421895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:15.421908 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:15.421982 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:15.422091 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.422883 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.424440 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.425169 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.425741 (+ 572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.425859 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:15.425886 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 13:21:15.425935 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":113,"threads_started":1}
I20260925 13:21:15.426137 20463 tablet_service.cc:1467] Processing CreateTablet for tablet 42a0e138810249b99d9aaa2e14d447da (DEFAULT_TABLE table=test-table [id=d679c90f758640d988be3fcb9b4ed6f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:15.426528 20463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a0e138810249b99d9aaa2e14d447da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:15.426607 20332 tablet_service.cc:1467] Processing CreateTablet for tablet 42a0e138810249b99d9aaa2e14d447da (DEFAULT_TABLE table=test-table [id=d679c90f758640d988be3fcb9b4ed6f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:15.426971 20332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a0e138810249b99d9aaa2e14d447da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:15.429239 20543 tablet_bootstrap.cc:492] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e: Bootstrap starting.
I20260925 13:21:15.429656 20544 tablet_bootstrap.cc:492] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081: Bootstrap starting.
I20260925 13:21:15.429944 20201 tablet_service.cc:1467] Processing CreateTablet for tablet 42a0e138810249b99d9aaa2e14d447da (DEFAULT_TABLE table=test-table [id=d679c90f758640d988be3fcb9b4ed6f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 13:21:15.430124 20543 tablet_bootstrap.cc:654] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:15.430243 20544 tablet_bootstrap.cc:654] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:15.430325 20201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a0e138810249b99d9aaa2e14d447da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:15.430446 20543 log.cc:824] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:15.430555 20544 log.cc:824] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:15.431250 20543 tablet_bootstrap.cc:492] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e: No bootstrap required, opened a new log
I20260925 13:21:15.431252 20544 tablet_bootstrap.cc:492] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081: No bootstrap required, opened a new log
I20260925 13:21:15.431340 20543 ts_tablet_manager.cc:1397] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 13:21:15.431362 20544 ts_tablet_manager.cc:1397] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 13:21:15.432986 20547 tablet_bootstrap.cc:492] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218: Bootstrap starting.
I20260925 13:21:15.433642 20547 tablet_bootstrap.cc:654] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:15.433908 20547 log.cc:824] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:15.433902 20543 raft_consensus.cc:348] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eb88bfa015496ea798a859da25e218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45847 } } peers { permanent_uuid: "84056f14ca49411ea762811d439f9081" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "fde8289cf0784c8e86163f2270068b8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44361 } }
I20260925 13:21:15.434085 20543 raft_consensus.cc:374] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:15.434124 20543 raft_consensus.cc:729] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde8289cf0784c8e86163f2270068b8e, State: Initialized, Role: FOLLOWER
I20260925 13:21:15.434260 20543 consensus_queue.cc:260] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eb88bfa015496ea798a859da25e218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45847 } } peers { permanent_uuid: "84056f14ca49411ea762811d439f9081" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "fde8289cf0784c8e86163f2270068b8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44361 } }
I20260925 13:21:15.434338 20544 raft_consensus.cc:348] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eb88bfa015496ea798a859da25e218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45847 } } peers { permanent_uuid: "84056f14ca49411ea762811d439f9081" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "fde8289cf0784c8e86163f2270068b8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44361 } }
I20260925 13:21:15.434491 20544 raft_consensus.cc:374] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:15.434530 20544 raft_consensus.cc:729] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84056f14ca49411ea762811d439f9081, State: Initialized, Role: FOLLOWER
I20260925 13:21:15.434541 20543 ts_tablet_manager.cc:1428] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 13:21:15.434614 20547 tablet_bootstrap.cc:492] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218: No bootstrap required, opened a new log
I20260925 13:21:15.434652 20529 heartbeater.cc:503] Master 127.0.98.62:45943 was elected leader, sending a full tablet report...
I20260925 13:21:15.434639 20544 consensus_queue.cc:260] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eb88bfa015496ea798a859da25e218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45847 } } peers { permanent_uuid: "84056f14ca49411ea762811d439f9081" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "fde8289cf0784c8e86163f2270068b8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44361 } }
I20260925 13:21:15.434682 20547 ts_tablet_manager.cc:1397] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 13:21:15.434881 20544 ts_tablet_manager.cc:1428] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 13:21:15.435123 20398 heartbeater.cc:503] Master 127.0.98.62:45943 was elected leader, sending a full tablet report...
I20260925 13:21:15.436707 20547 raft_consensus.cc:348] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eb88bfa015496ea798a859da25e218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45847 } } peers { permanent_uuid: "84056f14ca49411ea762811d439f9081" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "fde8289cf0784c8e86163f2270068b8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44361 } }
I20260925 13:21:15.436841 20547 raft_consensus.cc:374] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:15.436870 20547 raft_consensus.cc:729] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71eb88bfa015496ea798a859da25e218, State: Initialized, Role: FOLLOWER
I20260925 13:21:15.436988 20547 consensus_queue.cc:260] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eb88bfa015496ea798a859da25e218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45847 } } peers { permanent_uuid: "84056f14ca49411ea762811d439f9081" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "fde8289cf0784c8e86163f2270068b8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44361 } }
I20260925 13:21:15.437204 20547 ts_tablet_manager.cc:1428] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 13:21:15.437273 20267 heartbeater.cc:503] Master 127.0.98.62:45943 was elected leader, sending a full tablet report...
I20260925 13:21:15.513437 20551 raft_consensus.cc:482] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 13:21:15.513613 20551 raft_consensus.cc:504] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eb88bfa015496ea798a859da25e218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45847 } } peers { permanent_uuid: "84056f14ca49411ea762811d439f9081" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "fde8289cf0784c8e86163f2270068b8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44361 } }
I20260925 13:21:15.514020 20551 leader_election.cc:288] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84056f14ca49411ea762811d439f9081 (127.0.98.2:45179), fde8289cf0784c8e86163f2270068b8e (127.0.98.3:44361)
W20260925 13:21:15.515673 20399 tablet.cc:2369] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 13:21:15.516824 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.514203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45179 (local address 127.0.98.1:44473)
0925 13:21:15.514263 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:15.514274 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:15.514318 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:15.514542 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:15.514543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:15.514545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:15.514615 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.514617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.515258 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.515259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.516071 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.516073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.516603 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.516604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.516672 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.516676 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 13:21:15.516723 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 13:21:15.516768 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 13:21:15.516834 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.514335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44473 (local address 127.0.98.2:45179)
0925 13:21:15.514402 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:15.514404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:15.514412 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:21:15.514442 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:15.514443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:15.514452 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:15.514498 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:15.514670 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.515177 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.516156 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.516530 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.516581 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.516590 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:15.516735 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 13:21:15.516771 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:21:15.516942 20552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.514212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44361 (local address 127.0.98.1:48983)
0925 13:21:15.514397 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:15.514407 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:15.514463 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:15.514581 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:15.514582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:15.514584 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 13:21:15.514654 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.514656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.515259 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.515260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.516156 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:15.516159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.516703 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.516704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:15.516777 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.516780 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 13:21:15.516832 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 13:21:15.516870 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":14,"thread_start_us":95,"threads_started":1}
I20260925 13:21:15.516944 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.514391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48983 (local address 127.0.98.3:44361)
0925 13:21:15.514449 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:15.514451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:15.514459 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:21:15.514518 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:15.514519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:15.514527 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:15.514573 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 13:21:15.514719 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.515179 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.516240 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.516629 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.516694 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.516719 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:15.516844 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 13:21:15.516883 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:21:15.517181 20352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a0e138810249b99d9aaa2e14d447da" candidate_uuid: "71eb88bfa015496ea798a859da25e218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84056f14ca49411ea762811d439f9081" is_pre_election: true
I20260925 13:21:15.517267 20483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a0e138810249b99d9aaa2e14d447da" candidate_uuid: "71eb88bfa015496ea798a859da25e218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde8289cf0784c8e86163f2270068b8e" is_pre_election: true
I20260925 13:21:15.517370 20352 raft_consensus.cc:2454] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71eb88bfa015496ea798a859da25e218 in term 0.
I20260925 13:21:15.517433 20483 raft_consensus.cc:2454] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71eb88bfa015496ea798a859da25e218 in term 0.
I20260925 13:21:15.517622 20154 leader_election.cc:302] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71eb88bfa015496ea798a859da25e218, 84056f14ca49411ea762811d439f9081; no voters:
I20260925 13:21:15.517748 20551 raft_consensus.cc:2781] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 13:21:15.517803 20551 raft_consensus.cc:482] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 13:21:15.517820 20551 raft_consensus.cc:3037] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:15.518481 20551 raft_consensus.cc:504] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eb88bfa015496ea798a859da25e218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45847 } } peers { permanent_uuid: "84056f14ca49411ea762811d439f9081" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "fde8289cf0784c8e86163f2270068b8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44361 } }
I20260925 13:21:15.518597 20551 leader_election.cc:288] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [CANDIDATE]: Term 1 election: Requested vote from peers 84056f14ca49411ea762811d439f9081 (127.0.98.2:45179), fde8289cf0784c8e86163f2270068b8e (127.0.98.3:44361)
I20260925 13:21:15.518740 20352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a0e138810249b99d9aaa2e14d447da" candidate_uuid: "71eb88bfa015496ea798a859da25e218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84056f14ca49411ea762811d439f9081"
I20260925 13:21:15.518779 20483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a0e138810249b99d9aaa2e14d447da" candidate_uuid: "71eb88bfa015496ea798a859da25e218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde8289cf0784c8e86163f2270068b8e"
I20260925 13:21:15.518841 20483 raft_consensus.cc:3037] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:15.518828 20352 raft_consensus.cc:3037] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:15.519711 20483 raft_consensus.cc:2454] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eb88bfa015496ea798a859da25e218 in term 1.
I20260925 13:21:15.519711 20352 raft_consensus.cc:2454] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71eb88bfa015496ea798a859da25e218 in term 1.
I20260925 13:21:15.519918 20154 leader_election.cc:302] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71eb88bfa015496ea798a859da25e218, 84056f14ca49411ea762811d439f9081; no voters:
I20260925 13:21:15.520012 20551 raft_consensus.cc:2781] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:15.520207 20551 raft_consensus.cc:686] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [term 1 LEADER]: Becoming Leader. State: Replica: 71eb88bfa015496ea798a859da25e218, State: Running, Role: LEADER
I20260925 13:21:15.520330 20551 consensus_queue.cc:237] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eb88bfa015496ea798a859da25e218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45847 } } peers { permanent_uuid: "84056f14ca49411ea762811d439f9081" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "fde8289cf0784c8e86163f2270068b8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44361 } }
I20260925 13:21:15.521202 20078 catalog_manager.cc:5495] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71eb88bfa015496ea798a859da25e218 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "71eb88bfa015496ea798a859da25e218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71eb88bfa015496ea798a859da25e218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84056f14ca49411ea762811d439f9081" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fde8289cf0784c8e86163f2270068b8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44361 } health_report { overall_health: UNKNOWN } } }
I20260925 13:21:15.542191 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.539838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59184 (local address 127.0.98.1:45847)
0925 13:21:15.539885 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:15.539887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:15.539920 (+ 33us) server_negotiation.cc:407] Connection header received
0925 13:21:15.540006 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:15.540007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:15.540028 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:15.540087 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:15.540222 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.540693 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.541435 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:15.541644 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:15.541700 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:15.541852 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:15.541959 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:15.542032 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:15.542067 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 13:21:15.542091 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:21:15.543963 20483 raft_consensus.cc:1261] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e [term 1 FOLLOWER]: Refusing update from remote peer 71eb88bfa015496ea798a859da25e218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:15.543964 20352 raft_consensus.cc:1261] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081 [term 1 FOLLOWER]: Refusing update from remote peer 71eb88bfa015496ea798a859da25e218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:15.544293 20551 consensus_queue.cc:1035] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84056f14ca49411ea762811d439f9081" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:21:15.544427 20553 consensus_queue.cc:1035] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde8289cf0784c8e86163f2270068b8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:21:15.546542 20556 mvcc.cc:205] Tried to move back new op lower bound from 7333242779825553408 to 7333242779732008960. Current Snapshot: MvccSnapshot[applied={T|T < 7333242779825553408}]
I20260925 13:21:15.547216 20557 mvcc.cc:205] Tried to move back new op lower bound from 7333242779825553408 to 7333242779732008960. Current Snapshot: MvccSnapshot[applied={T|T < 7333242779825553408}]
I20260925 13:21:15.548879 20078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:21:15.548960 20078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 13:21:15.549700 20078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3afe58d10dde43ffa2d756179acbda50: Sending DeleteTablet for 3 replicas of tablet 42a0e138810249b99d9aaa2e14d447da
I20260925 13:21:15.549969 20201 tablet_service.cc:1514] Processing DeleteTablet for tablet 42a0e138810249b99d9aaa2e14d447da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:15 UTC) from {username='slave'} at 127.0.0.1:59168
I20260925 13:21:15.550072 20332 tablet_service.cc:1514] Processing DeleteTablet for tablet 42a0e138810249b99d9aaa2e14d447da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:15 UTC) from {username='slave'} at 127.0.0.1:41764
I20260925 13:21:15.550177 20566 tablet_replica.cc:330] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218: stopping tablet replica
I20260925 13:21:15.550199 20463 tablet_service.cc:1514] Processing DeleteTablet for tablet 42a0e138810249b99d9aaa2e14d447da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 13:21:15 UTC) from {username='slave'} at 127.0.0.1:35372
I20260925 13:21:15.550266 20566 raft_consensus.cc:2229] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [term 1 LEADER]: Raft consensus shutting down.
I20260925 13:21:15.550371 20566 raft_consensus.cc:2258] T 42a0e138810249b99d9aaa2e14d447da P 71eb88bfa015496ea798a859da25e218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:15.550601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 20139
I20260925 13:21:15.553638 20568 tablet_replica.cc:330] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e: stopping tablet replica
I20260925 13:21:15.553736 20568 raft_consensus.cc:2229] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:21:15.553821 20568 raft_consensus.cc:2258] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:15.553963 20567 tablet_replica.cc:330] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081: stopping tablet replica
I20260925 13:21:15.554070 20567 raft_consensus.cc:2229] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 13:21:15.554172 20567 raft_consensus.cc:2258] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 13:21:15.554291 20568 ts_tablet_manager.cc:1905] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 13:21:15.554630 20567 ts_tablet_manager.cc:1905] T 42a0e138810249b99d9aaa2e14d447da P 84056f14ca49411ea762811d439f9081: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 13:21:15.555429 20065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45847 (error 108)
I20260925 13:21:15.555627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 20270
I20260925 13:21:15.556152 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.555769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45847 (local address 127.0.0.1:59194)
0925 13:21:15.556081 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:15.556119 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
W20260925 13:21:15.556262 20065 catalog_manager.cc:4584] TS 71eb88bfa015496ea798a859da25e218 (127.0.98.1:45847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42a0e138810249b99d9aaa2e14d447da: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45847: connect: Connection refused (error 111)
I20260925 13:21:15.557420 20568 ts_tablet_manager.cc:1918] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 13:21:15.557493 20568 log.cc:1196] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e: Deleting WAL directory at /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790342239834770-392-0/minicluster-data/ts-2/wal/wals/42a0e138810249b99d9aaa2e14d447da
I20260925 13:21:15.557737 20568 ts_tablet_manager.cc:1939] T 42a0e138810249b99d9aaa2e14d447da P fde8289cf0784c8e86163f2270068b8e: Deleting consensus metadata
I20260925 13:21:15.558158 20066 catalog_manager.cc:4841] TS fde8289cf0784c8e86163f2270068b8e (127.0.98.3:44361): tablet 42a0e138810249b99d9aaa2e14d447da (table test-table [id=d679c90f758640d988be3fcb9b4ed6f2]) successfully deleted
I20260925 13:21:15.560287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 20401
I20260925 13:21:15.560359 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:15.560256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45179 (local address 127.0.0.1:41776)
0925 13:21:15.560299 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:15.560324 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 13:21:15.564623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 20048
2026-09-25T13:21:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (734 ms)
[----------] 6 tests from EncryptionPolicyTest (8167 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:15 dist-test-slave-t8d9 krb5kdc[20572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:15 dist-test-slave-t8d9 krb5kdc[20572](info): set up 2 sockets
Sep 25 13:21:15 dist-test-slave-t8d9 krb5kdc[20572](info): commencing operation
krb5kdc: starting...
W20260925 13:21:16.517186 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:16 dist-test-slave-t8d9 krb5kdc[20572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:21:16.628216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:16.708490 20585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:16.708719 20585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:16.708742 20585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:16.709924 20585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:16.709966 20585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:16.709980 20585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:16.709991 20585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:16.710001 20585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:16.711481 20585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:16.711691 20585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:16.711870 20585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:16.716547 20591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:16.716703 20585 server_base.cc:1034] running on GCE node
W20260925 13:21:16.716557 20593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:16.716565 20590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:16.717020 20585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:16.717305 20585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:16.718472 20585 hybrid_clock.cc:648] HybridClock initialized: now 1790342476718456 us; error 28 us; skew 500 ppm
Sep 25 13:21:16 dist-test-slave-t8d9 krb5kdc[20572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:16.720806 20585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:21:16.721220 20585 webserver.cc:460] Webserver started at http://127.0.0.1:34425/ using document root <none> and password file <none>
I20260925 13:21:16.721434 20585 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:16.721480 20585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:16.721616 20585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:16.722697 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "f25d3d4a60cc4f4bac2380510507b1f9"
format_stamp: "Formatted at 2026-09-25 13:21:16 on dist-test-slave-t8d9"
server_key: "448f9c3c27d817dc3a1d944d06267b33"
server_key_iv: "08224bd3f3a601402f1c0fc419f4782f"
server_key_version: "encryptionkey@0"
I20260925 13:21:16.723031 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "f25d3d4a60cc4f4bac2380510507b1f9"
format_stamp: "Formatted at 2026-09-25 13:21:16 on dist-test-slave-t8d9"
server_key: "448f9c3c27d817dc3a1d944d06267b33"
server_key_iv: "08224bd3f3a601402f1c0fc419f4782f"
server_key_version: "encryptionkey@0"
I20260925 13:21:16.724262 20585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:21:16.724983 20600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:16.725153 20585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:16.725234 20585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "f25d3d4a60cc4f4bac2380510507b1f9"
format_stamp: "Formatted at 2026-09-25 13:21:16 on dist-test-slave-t8d9"
server_key: "448f9c3c27d817dc3a1d944d06267b33"
server_key_iv: "08224bd3f3a601402f1c0fc419f4782f"
server_key_version: "encryptionkey@0"
I20260925 13:21:16.725296 20585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:16.742321 20585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:16.744249 20585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:16.744385 20585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:16.748091 20585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46305
I20260925 13:21:16.748138 20652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46305 every 8 connection(s)
I20260925 13:21:16.748466 20585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:16.749333 20653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:16.751565 20653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9: Bootstrap starting.
I20260925 13:21:16.752061 20653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:16.752278 20653 log.cc:824] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:16.752882 20653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9: No bootstrap required, opened a new log
I20260925 13:21:16.753275 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 20585
I20260925 13:21:16.753360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:16.753527 392 external_mini_cluster.cc:1428] Setting key 6ea5b6160df23df61037be672c0c5119
I20260925 13:21:16.754808 20653 raft_consensus.cc:348] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25d3d4a60cc4f4bac2380510507b1f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46305 } }
I20260925 13:21:16.754961 20653 raft_consensus.cc:374] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:16.754997 20653 raft_consensus.cc:729] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f25d3d4a60cc4f4bac2380510507b1f9, State: Initialized, Role: FOLLOWER
I20260925 13:21:16.755152 20653 consensus_queue.cc:260] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25d3d4a60cc4f4bac2380510507b1f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46305 } }
I20260925 13:21:16.755245 20653 raft_consensus.cc:388] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:16.755277 20653 raft_consensus.cc:482] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:16.755343 20653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:16.755993 20653 raft_consensus.cc:504] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25d3d4a60cc4f4bac2380510507b1f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46305 } }
I20260925 13:21:16.756083 20653 leader_election.cc:302] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f25d3d4a60cc4f4bac2380510507b1f9; no voters:
I20260925 13:21:16.756245 20653 leader_election.cc:288] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:16.756286 20658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:16.756443 20658 raft_consensus.cc:686] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [term 1 LEADER]: Becoming Leader. State: Replica: f25d3d4a60cc4f4bac2380510507b1f9, State: Running, Role: LEADER
I20260925 13:21:16.756508 20653 sys_catalog.cc:564] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:16.756580 20658 consensus_queue.cc:237] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25d3d4a60cc4f4bac2380510507b1f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46305 } }
I20260925 13:21:16.756889 20658 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f25d3d4a60cc4f4bac2380510507b1f9. Latest consensus state: current_term: 1 leader_uuid: "f25d3d4a60cc4f4bac2380510507b1f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25d3d4a60cc4f4bac2380510507b1f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46305 } } }
I20260925 13:21:16.757130 20658 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:16.757061 20660 sys_catalog.cc:455] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f25d3d4a60cc4f4bac2380510507b1f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f25d3d4a60cc4f4bac2380510507b1f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46305 } } }
I20260925 13:21:16.757555 20660 sys_catalog.cc:458] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:16.757561 20667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:16.758208 20667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:16.760294 20667 catalog_manager.cc:1347] Generated new cluster ID: c3a6955e9a394b959bc70ea16def7ccb
I20260925 13:21:16.760336 20667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:21:16 dist-test-slave-t8d9 krb5kdc[20572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 13:21:16.764958 20657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:16.754065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47285 (local address 127.0.0.1:46305)
0925 13:21:16.754203 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:16.754211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:16.754238 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:21:16.754918 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:16.754934 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:16.754985 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:16.755118 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:16.755895 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:16.757116 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:16.758236 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:16.758797 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:16.758903 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:16.761515 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:16.761519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:16.761519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:16.761549 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:16.763356 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:16.763802 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:16.763805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:16.763805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:16.763899 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:16.764124 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:16.764124 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:16.764125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:16.764389 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:16.764535 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:16.764663 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 13:21:16.764757 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260925 13:21:16.766022 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:16.766381 20667 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:16.766798 20667 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:16.771510 20667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f25d3d4a60cc4f4bac2380510507b1f9: Generated new TSK 0
I20260925 13:21:16.771677 20667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:16.774212 20657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:16.767445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56947 (local address 127.0.0.1:46305)
0925 13:21:16.767481 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:16.767482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:16.767577 (+ 95us) server_negotiation.cc:407] Connection header received
0925 13:21:16.767618 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:16.767619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:16.767628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:16.767691 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:16.768234 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:16.769025 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:16.770011 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:16.770341 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:16.770407 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:16.771834 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:16.771835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:16.771836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:16.771843 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:16.773207 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:16.773521 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:16.773522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:16.773522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:16.773596 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:16.773773 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:16.773773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:16.773774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:16.773892 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:16.773997 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:16.774063 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 13:21:16.774123 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 13:21:16.776162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 20585
W20260925 13:21:16.780050 20570 connection.cc:535] client connection to 127.0.0.1:46305 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46305): Connection reset by peer (error 104)
2026-09-25T13:21:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:16 dist-test-slave-t8d9 krb5kdc[20684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:16 dist-test-slave-t8d9 krb5kdc[20684](info): set up 2 sockets
Sep 25 13:21:16 dist-test-slave-t8d9 krb5kdc[20684](info): commencing operation
krb5kdc: starting...
W20260925 13:21:17.717207 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:17 dist-test-slave-t8d9 krb5kdc[20684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:21:17.832206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:17.910775 20697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:17.910993 20697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:17.911016 20697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:17.912133 20697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:17.912168 20697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:17.912180 20697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:17.912191 20697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:17.912201 20697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:17.913669 20697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45129
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:17.913882 20697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:17.914068 20697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:17.918792 20702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:17.918821 20697 server_base.cc:1034] running on GCE node
W20260925 13:21:17.918851 20703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:17.918771 20705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:17.919251 20697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:17.919531 20697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:17.920722 20697 hybrid_clock.cc:648] HybridClock initialized: now 1790342477920701 us; error 34 us; skew 500 ppm
Sep 25 13:21:17 dist-test-slave-t8d9 krb5kdc[20684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:17.927356 20697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:21:17.927875 20697 webserver.cc:460] Webserver started at http://127.0.0.1:42843/ using document root <none> and password file <none>
I20260925 13:21:17.928104 20697 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:17.928155 20697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:17.928282 20697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:17.929382 20697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "d40021cd01124f1aa1b4caaf16fd3250"
format_stamp: "Formatted at 2026-09-25 13:21:17 on dist-test-slave-t8d9"
server_key: "db24bac40e124e2c333b86ba14bf359d"
server_key_iv: "53ccf57bf15e0bda580848e0fd2a877d"
server_key_version: "encryptionkey@0"
I20260925 13:21:17.929733 20697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "d40021cd01124f1aa1b4caaf16fd3250"
format_stamp: "Formatted at 2026-09-25 13:21:17 on dist-test-slave-t8d9"
server_key: "db24bac40e124e2c333b86ba14bf359d"
server_key_iv: "53ccf57bf15e0bda580848e0fd2a877d"
server_key_version: "encryptionkey@0"
I20260925 13:21:17.930990 20697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:21:17.931680 20712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:17.931877 20697 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:21:17.931982 20697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "d40021cd01124f1aa1b4caaf16fd3250"
format_stamp: "Formatted at 2026-09-25 13:21:17 on dist-test-slave-t8d9"
server_key: "db24bac40e124e2c333b86ba14bf359d"
server_key_iv: "53ccf57bf15e0bda580848e0fd2a877d"
server_key_version: "encryptionkey@0"
I20260925 13:21:17.932050 20697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:17.945719 20697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:17.947795 20697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:17.947942 20697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:17.951748 20697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45129
I20260925 13:21:17.951815 20764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45129 every 8 connection(s)
I20260925 13:21:17.952224 20697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:17.953094 20765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:17.955402 20765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250: Bootstrap starting.
I20260925 13:21:17.955965 20765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:17.956218 20765 log.cc:824] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:17.956830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 20697
I20260925 13:21:17.956931 20765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250: No bootstrap required, opened a new log
I20260925 13:21:17.956957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:17.957156 392 external_mini_cluster.cc:1428] Setting key f10e90ee243864061911ac903e951fb7
I20260925 13:21:17.958814 20765 raft_consensus.cc:348] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40021cd01124f1aa1b4caaf16fd3250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } }
I20260925 13:21:17.958954 20765 raft_consensus.cc:374] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:17.958986 20765 raft_consensus.cc:729] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40021cd01124f1aa1b4caaf16fd3250, State: Initialized, Role: FOLLOWER
I20260925 13:21:17.959128 20765 consensus_queue.cc:260] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40021cd01124f1aa1b4caaf16fd3250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } }
I20260925 13:21:17.959203 20765 raft_consensus.cc:388] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:17.959233 20765 raft_consensus.cc:482] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:17.959280 20765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:17.959786 20765 raft_consensus.cc:504] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40021cd01124f1aa1b4caaf16fd3250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } }
I20260925 13:21:17.959880 20765 leader_election.cc:302] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d40021cd01124f1aa1b4caaf16fd3250; no voters:
I20260925 13:21:17.960031 20765 leader_election.cc:288] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:17.960064 20770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:17.960210 20770 raft_consensus.cc:686] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [term 1 LEADER]: Becoming Leader. State: Replica: d40021cd01124f1aa1b4caaf16fd3250, State: Running, Role: LEADER
I20260925 13:21:17.960270 20765 sys_catalog.cc:564] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:17.960346 20770 consensus_queue.cc:237] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40021cd01124f1aa1b4caaf16fd3250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } }
I20260925 13:21:17.960749 20771 sys_catalog.cc:455] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d40021cd01124f1aa1b4caaf16fd3250. Latest consensus state: current_term: 1 leader_uuid: "d40021cd01124f1aa1b4caaf16fd3250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40021cd01124f1aa1b4caaf16fd3250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } } }
I20260925 13:21:17.960877 20771 sys_catalog.cc:458] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:17.961830 20772 sys_catalog.cc:455] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d40021cd01124f1aa1b4caaf16fd3250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40021cd01124f1aa1b4caaf16fd3250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45129 } } }
W20260925 13:21:17.961948 20786 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 13:21:17.961956 20772 sys_catalog.cc:458] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250 [sys.catalog]: This master's current role is: LEADER
W20260925 13:21:17.962122 20786 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:21:17.962216 20779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:17.962404 20779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:17.963800 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:17.957686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42417 (local address 127.0.0.1:45129)
0925 13:21:17.957869 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:17.957878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:17.957912 (+ 34us) server_negotiation.cc:407] Connection header received
0925 13:21:17.958783 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:17.958790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:17.958838 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:17.958961 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:17.959171 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:17.960602 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:17.961682 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:17.962277 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:17.962375 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:17.963038 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:17.963042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:17.963042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:17.963233 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:17.963271 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:17.963285 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:17.963369 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:17.963537 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 13:21:17.963568 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260925 13:21:17.964063 20779 catalog_manager.cc:1347] Generated new cluster ID: 9feafaa2178141cc8506814f1dc97ac4
I20260925 13:21:17.964111 20779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:17.964577 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:17.969363 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:17.965973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41071 (local address 127.0.0.1:45129)
0925 13:21:17.966006 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:17.966007 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:17.966110 (+ 103us) server_negotiation.cc:407] Connection header received
0925 13:21:17.966185 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:17.966186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:17.966195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:17.966220 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:17.966379 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:17.967118 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:17.968082 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:17.968381 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:17.968415 (+ 34us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:17.969128 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:17.969129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:17.969129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:17.969134 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:17.969146 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:17.969150 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:17.969176 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:17.969275 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 13:21:17.969286 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 13:21:17.975735 20779 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:17.976240 20779 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:17.983327 20779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d40021cd01124f1aa1b4caaf16fd3250: Generated new TSK 0
I20260925 13:21:17.983500 20779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:17.987149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 20697
W20260925 13:21:17.991489 20682 connection.cc:535] client connection to 127.0.0.1:45129 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45129): Connection reset by peer (error 104)
2026-09-25T13:21:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:18 dist-test-slave-t8d9 krb5kdc[20796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:18 dist-test-slave-t8d9 krb5kdc[20796](info): set up 2 sockets
Sep 25 13:21:18 dist-test-slave-t8d9 krb5kdc[20796](info): commencing operation
krb5kdc: starting...
W20260925 13:21:18.953219 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:19 dist-test-slave-t8d9 krb5kdc[20796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:21:19.066146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:19.145474 20809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:19.145740 20809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:19.145762 20809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:19.146883 20809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:19.146917 20809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:19.146929 20809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:19.146940 20809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:19.146956 20809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:19.148355 20809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42313
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:19.148548 20809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:19.148715 20809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:19.153234 20817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:19.153210 20815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:19.153264 20814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:19.153462 20809 server_base.cc:1034] running on GCE node
I20260925 13:21:19.153708 20809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:19.153970 20809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:19.155099 20809 hybrid_clock.cc:648] HybridClock initialized: now 1790342479155090 us; error 27 us; skew 500 ppm
Sep 25 13:21:19 dist-test-slave-t8d9 krb5kdc[20796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:19.157289 20809 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:21:19.157737 20809 webserver.cc:460] Webserver started at http://127.0.0.1:34451/ using document root <none> and password file <none>
I20260925 13:21:19.157940 20809 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:19.157987 20809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:19.158103 20809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:19.159122 20809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "ea3e89b61d5d444da32da503fc50e933"
format_stamp: "Formatted at 2026-09-25 13:21:19 on dist-test-slave-t8d9"
server_key: "73b39c70f7b6463f70871172c97906d1"
server_key_iv: "48e05d75a47db14e49ac9871507c21d5"
server_key_version: "encryptionkey@0"
I20260925 13:21:19.159435 20809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea3e89b61d5d444da32da503fc50e933"
format_stamp: "Formatted at 2026-09-25 13:21:19 on dist-test-slave-t8d9"
server_key: "73b39c70f7b6463f70871172c97906d1"
server_key_iv: "48e05d75a47db14e49ac9871507c21d5"
server_key_version: "encryptionkey@0"
I20260925 13:21:19.160626 20809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:21:19.161356 20824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:19.161504 20809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:19.161615 20809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "ea3e89b61d5d444da32da503fc50e933"
format_stamp: "Formatted at 2026-09-25 13:21:19 on dist-test-slave-t8d9"
server_key: "73b39c70f7b6463f70871172c97906d1"
server_key_iv: "48e05d75a47db14e49ac9871507c21d5"
server_key_version: "encryptionkey@0"
I20260925 13:21:19.161680 20809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:19.164898 20809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:19.166998 20809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:19.167193 20809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:19.170760 20809 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42313
I20260925 13:21:19.170799 20876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42313 every 8 connection(s)
I20260925 13:21:19.171078 20809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:19.171918 20877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:19.174095 20877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933: Bootstrap starting.
I20260925 13:21:19.174608 20877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:19.174850 20877 log.cc:824] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:19.175419 20877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933: No bootstrap required, opened a new log
I20260925 13:21:19.176923 20877 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e89b61d5d444da32da503fc50e933" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42313 } }
I20260925 13:21:19.177059 20877 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:19.177080 20877 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3e89b61d5d444da32da503fc50e933, State: Initialized, Role: FOLLOWER
I20260925 13:21:19.177208 20877 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e89b61d5d444da32da503fc50e933" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42313 } }
I20260925 13:21:19.177265 20877 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:19.177284 20877 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:19.177325 20877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:19.177861 20877 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e89b61d5d444da32da503fc50e933" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42313 } }
I20260925 13:21:19.177945 20877 leader_election.cc:302] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea3e89b61d5d444da32da503fc50e933; no voters:
I20260925 13:21:19.178087 20877 leader_election.cc:288] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:19.178144 20880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:19.178300 20877 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:19.178453 20880 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [term 1 LEADER]: Becoming Leader. State: Replica: ea3e89b61d5d444da32da503fc50e933, State: Running, Role: LEADER
I20260925 13:21:19.178550 20880 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e89b61d5d444da32da503fc50e933" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42313 } }
I20260925 13:21:19.178886 20881 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea3e89b61d5d444da32da503fc50e933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e89b61d5d444da32da503fc50e933" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42313 } } }
I20260925 13:21:19.178982 20881 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:19.178905 20882 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea3e89b61d5d444da32da503fc50e933. Latest consensus state: current_term: 1 leader_uuid: "ea3e89b61d5d444da32da503fc50e933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e89b61d5d444da32da503fc50e933" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42313 } } }
I20260925 13:21:19.179171 20882 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:19.179178 20889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:19.179740 20889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:19.180521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 20809
I20260925 13:21:19.180636 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:19.180820 392 external_mini_cluster.cc:1428] Setting key 5999b65add9c6c155aad3b58e3532cfb
I20260925 13:21:19.181282 20889 catalog_manager.cc:1347] Generated new cluster ID: f71adde80d4c4071904b6a66d0ce4659
I20260925 13:21:19.181326 20889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:19.183126 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:19.181230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58785 (local address 127.0.0.1:42313)
0925 13:21:19.181659 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:19.181669 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:19.181864 (+ 195us) server_negotiation.cc:407] Connection header received
0925 13:21:19.181990 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:19.181997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:19.182028 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:19.182161 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:19.182757 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:19.182760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:19.182760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:19.182764 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:19.182803 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:19.182816 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:19.182864 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:19.182929 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 13:21:19.182964 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":76,"threads_started":1}
I20260925 13:21:19.183882 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:19.186239 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:19.185240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37853 (local address 127.0.0.1:42313)
0925 13:21:19.185270 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:19.185271 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:19.185397 (+ 126us) server_negotiation.cc:407] Connection header received
0925 13:21:19.185444 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:19.185445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:19.185449 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:19.185501 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:19.186025 (+ 524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:19.186026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:19.186026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:19.186037 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:19.186050 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:19.186054 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:19.186108 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:19.186167 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 13:21:19.186179 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 13:21:19.187407 20889 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:19.187546 20889 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:19.193764 20889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea3e89b61d5d444da32da503fc50e933: Generated new TSK 0
I20260925 13:21:19.193928 20889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:19.195773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 20809
2026-09-25T13:21:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:19 dist-test-slave-t8d9 krb5kdc[20908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:19 dist-test-slave-t8d9 krb5kdc[20908](info): set up 2 sockets
Sep 25 13:21:19 dist-test-slave-t8d9 krb5kdc[20908](info): commencing operation
krb5kdc: starting...
W20260925 13:21:20.141275 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:20 dist-test-slave-t8d9 krb5kdc[20908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:21:20.255748 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:20.337097 20921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:20.337327 20921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:20.337361 20921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:20.338683 20921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:20.338739 20921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:20.338757 20921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:20.338773 20921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:20.338789 20921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:20.340428 20921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:20.340698 20921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:20.340920 20921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:20.345721 20926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:20.345727 20929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:20.345727 20927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:20.345970 20921 server_base.cc:1034] running on GCE node
I20260925 13:21:20.346261 20921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:20.346553 20921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:20.347724 20921 hybrid_clock.cc:648] HybridClock initialized: now 1790342480347707 us; error 28 us; skew 500 ppm
Sep 25 13:21:20 dist-test-slave-t8d9 krb5kdc[20908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:20.350274 20921 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:21:20.350735 20921 webserver.cc:460] Webserver started at http://127.0.0.1:46043/ using document root <none> and password file <none>
I20260925 13:21:20.350963 20921 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:20.351012 20921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:20.351140 20921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:20.352245 20921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "070ac1f5860d4d33bf77030aa5f058aa"
format_stamp: "Formatted at 2026-09-25 13:21:20 on dist-test-slave-t8d9"
server_key: "f420f98b360c00e0b338b02f832e5695"
server_key_iv: "2bcb7192d854d44296fa647d45737bf1"
server_key_version: "encryptionkey@0"
I20260925 13:21:20.352594 20921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "070ac1f5860d4d33bf77030aa5f058aa"
format_stamp: "Formatted at 2026-09-25 13:21:20 on dist-test-slave-t8d9"
server_key: "f420f98b360c00e0b338b02f832e5695"
server_key_iv: "2bcb7192d854d44296fa647d45737bf1"
server_key_version: "encryptionkey@0"
I20260925 13:21:20.353832 20921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:21:20.354534 20936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:20.354707 20921 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:21:20.354784 20921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "070ac1f5860d4d33bf77030aa5f058aa"
format_stamp: "Formatted at 2026-09-25 13:21:20 on dist-test-slave-t8d9"
server_key: "f420f98b360c00e0b338b02f832e5695"
server_key_iv: "2bcb7192d854d44296fa647d45737bf1"
server_key_version: "encryptionkey@0"
I20260925 13:21:20.354844 20921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:20.362668 20921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:20.364722 20921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:20.364858 20921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:20.368427 20921 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35371
I20260925 13:21:20.368474 20988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35371 every 8 connection(s)
I20260925 13:21:20.368783 20921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:20.369645 20989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:20.370581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 20921
I20260925 13:21:20.370688 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:20.370867 392 external_mini_cluster.cc:1428] Setting key de0ad3a11c262aca99129a05a9047cbf
I20260925 13:21:20.372642 20989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa: Bootstrap starting.
I20260925 13:21:20.373265 20989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:20.373613 20989 log.cc:824] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:20.374356 20989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa: No bootstrap required, opened a new log
I20260925 13:21:20.376533 20989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ac1f5860d4d33bf77030aa5f058aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35371 } }
I20260925 13:21:20.376686 20989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:20.376718 20989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070ac1f5860d4d33bf77030aa5f058aa, State: Initialized, Role: FOLLOWER
I20260925 13:21:20.376848 20989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ac1f5860d4d33bf77030aa5f058aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35371 } }
I20260925 13:21:20.376922 20989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:20.376955 20989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:20.377002 20989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:20.377605 20989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ac1f5860d4d33bf77030aa5f058aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35371 } }
I20260925 13:21:20.377710 20989 leader_election.cc:302] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 070ac1f5860d4d33bf77030aa5f058aa; no voters:
I20260925 13:21:20.377889 20989 leader_election.cc:288] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:20.377930 20994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:20.378078 20994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [term 1 LEADER]: Becoming Leader. State: Replica: 070ac1f5860d4d33bf77030aa5f058aa, State: Running, Role: LEADER
I20260925 13:21:20.378189 20989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:20.378211 20994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ac1f5860d4d33bf77030aa5f058aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35371 } }
Sep 25 13:21:20 dist-test-slave-t8d9 krb5kdc[20908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 13:21:20.378582 20994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070ac1f5860d4d33bf77030aa5f058aa. Latest consensus state: current_term: 1 leader_uuid: "070ac1f5860d4d33bf77030aa5f058aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ac1f5860d4d33bf77030aa5f058aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35371 } } }
I20260925 13:21:20.378654 20995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "070ac1f5860d4d33bf77030aa5f058aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070ac1f5860d4d33bf77030aa5f058aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35371 } } }
I20260925 13:21:20.378809 20995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:20.379052 21003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:20.378720 20994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:20.379592 21003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:20.382207 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:20.371476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56131 (local address 127.0.0.1:35371)
0925 13:21:20.371675 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:20.371686 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:20.371719 (+ 33us) server_negotiation.cc:407] Connection header received
0925 13:21:20.372328 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:20.372338 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:20.372406 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:20.372576 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:20.373270 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:20.374598 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:20.375352 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:20.376130 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:20.376261 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:20.378846 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:20.378851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:20.378852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:20.378865 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:20.380659 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:20.381062 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:20.381065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:20.381065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:20.381145 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:20.381337 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:20.381337 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:20.381337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:20.381597 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:20.381740 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:20.381880 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 13:21:20.382017 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260925 13:21:20.384019 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:20.389642 21003 catalog_manager.cc:1347] Generated new cluster ID: 76a3dfb6402244d0ab9aaa3a541f316a
I20260925 13:21:20.389714 21003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:20.392437 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:20.385523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40819 (local address 127.0.0.1:35371)
0925 13:21:20.385615 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:20.385618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:20.385681 (+ 63us) server_negotiation.cc:407] Connection header received
0925 13:21:20.385728 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:20.385729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:20.385740 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:20.385798 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:20.386275 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:20.387036 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:20.387801 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:20.388143 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:20.388214 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:20.389800 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:20.389802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:20.389802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:20.389810 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:20.391283 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:20.391657 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:20.391658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:20.391658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:20.391721 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:20.391907 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:20.391908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:20.391909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:20.392034 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:20.392129 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:20.392256 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 13:21:20.392346 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 13:21:20.398772 21003 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:20.399356 21003 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:20.403277 21003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 070ac1f5860d4d33bf77030aa5f058aa: Generated new TSK 0
I20260925 13:21:20.403506 21003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:20.407431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 20921
2026-09-25T13:21:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:20 dist-test-slave-t8d9 krb5kdc[21020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:20 dist-test-slave-t8d9 krb5kdc[21020](info): set up 2 sockets
Sep 25 13:21:20 dist-test-slave-t8d9 krb5kdc[21020](info): commencing operation
krb5kdc: starting...
W20260925 13:21:21.353368 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:21 dist-test-slave-t8d9 krb5kdc[21020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:21:21.467108 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:21.545876 21033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:21.546088 21033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:21.546119 21033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:21.547267 21033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:21.547309 21033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:21.547326 21033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:21.547343 21033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:21.547358 21033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:21.548808 21033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:21.549057 21033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:21.549263 21033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:21.553833 21041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:21.553848 21039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:21.553952 21033 server_base.cc:1034] running on GCE node
W20260925 13:21:21.553844 21038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:21.554320 21033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:21.554636 21033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:21.555786 21033 hybrid_clock.cc:648] HybridClock initialized: now 1790342481555712 us; error 93 us; skew 500 ppm
Sep 25 13:21:21 dist-test-slave-t8d9 krb5kdc[21020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:21.558271 21033 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:21:21.558687 21033 webserver.cc:460] Webserver started at http://127.0.0.1:37971/ using document root <none> and password file <none>
I20260925 13:21:21.558863 21033 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:21.558902 21033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:21.558991 21033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:21.559993 21033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "4c451699534345628bb1e150d2367949"
format_stamp: "Formatted at 2026-09-25 13:21:21 on dist-test-slave-t8d9"
server_key: "6244a18269b562bfb0d82d526bd14699"
server_key_iv: "7b4bf110101ddad1f5304c5df82eace4"
server_key_version: "encryptionkey@0"
I20260925 13:21:21.560344 21033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c451699534345628bb1e150d2367949"
format_stamp: "Formatted at 2026-09-25 13:21:21 on dist-test-slave-t8d9"
server_key: "6244a18269b562bfb0d82d526bd14699"
server_key_iv: "7b4bf110101ddad1f5304c5df82eace4"
server_key_version: "encryptionkey@0"
I20260925 13:21:21.561587 21033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:21:21.562386 21048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:21.562577 21033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:21:21.562670 21033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "4c451699534345628bb1e150d2367949"
format_stamp: "Formatted at 2026-09-25 13:21:21 on dist-test-slave-t8d9"
server_key: "6244a18269b562bfb0d82d526bd14699"
server_key_iv: "7b4bf110101ddad1f5304c5df82eace4"
server_key_version: "encryptionkey@0"
I20260925 13:21:21.562737 21033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:21.573807 21033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:21.575897 21033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:21.576012 21033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:21.579743 21033 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44931
I20260925 13:21:21.579795 21100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44931 every 8 connection(s)
I20260925 13:21:21.580111 21033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:21.580974 21101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:21.581694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 21033
I20260925 13:21:21.581810 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:21.581979 392 external_mini_cluster.cc:1428] Setting key 486e8ba8439f48959af2077841fb6cb3
I20260925 13:21:21.584089 21101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949: Bootstrap starting.
I20260925 13:21:21.584803 21101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:21.585103 21101 log.cc:824] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:21.585805 21101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949: No bootstrap required, opened a new log
I20260925 13:21:21.588048 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:21.582516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44635 (local address 127.0.0.1:44931)
0925 13:21:21.582678 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:21.582688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:21.582718 (+ 30us) server_negotiation.cc:407] Connection header received
0925 13:21:21.583247 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:21.583257 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:21.583347 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:21.583512 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:21.583718 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:21.585017 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:21.585940 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:21.586565 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:21.586681 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:21.587348 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:21.587353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:21.587353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:21.587562 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:21.587606 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:21.587622 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:21.587694 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:21.587759 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 13:21:21.587832 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260925 13:21:21.588207 21101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c451699534345628bb1e150d2367949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44931 } }
I20260925 13:21:21.588344 21101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:21.588364 21101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c451699534345628bb1e150d2367949, State: Initialized, Role: FOLLOWER
I20260925 13:21:21.588487 21101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c451699534345628bb1e150d2367949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44931 } }
I20260925 13:21:21.588562 21101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:21.588599 21101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:21.588645 21101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:21.589042 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:21.589326 21101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c451699534345628bb1e150d2367949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44931 } }
I20260925 13:21:21.589429 21101 leader_election.cc:302] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c451699534345628bb1e150d2367949; no voters:
I20260925 13:21:21.589615 21101 leader_election.cc:288] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:21.589665 21106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:21.589812 21106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [term 1 LEADER]: Becoming Leader. State: Replica: 4c451699534345628bb1e150d2367949, State: Running, Role: LEADER
I20260925 13:21:21.589896 21101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:21.589942 21106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c451699534345628bb1e150d2367949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44931 } }
I20260925 13:21:21.590356 21106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c451699534345628bb1e150d2367949. Latest consensus state: current_term: 1 leader_uuid: "4c451699534345628bb1e150d2367949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c451699534345628bb1e150d2367949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44931 } } }
I20260925 13:21:21.590387 21108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c451699534345628bb1e150d2367949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c451699534345628bb1e150d2367949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44931 } } }
I20260925 13:21:21.590509 21108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:21.590720 21120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:21.590723 21106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:21.591303 21120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:21.593694 21120 catalog_manager.cc:1347] Generated new cluster ID: 8d9f89d9cdd04ab5982e26bdb8efaf19
I20260925 13:21:21.593745 21120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:21.594450 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:21.590535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49771 (local address 127.0.0.1:44931)
0925 13:21:21.590568 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:21.590570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:21.590958 (+ 388us) server_negotiation.cc:407] Connection header received
0925 13:21:21.591290 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:21.591291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:21.591302 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:21.591331 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:21.591534 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:21.592234 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:21.593199 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:21.593402 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:21.593468 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:21.594206 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:21.594208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:21.594208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:21.594214 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:21.594230 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:21.594235 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:21.594278 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:21.594338 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 13:21:21.594364 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 13:21:21.600844 21120 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:21.601372 21120 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:21.606155 21120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c451699534345628bb1e150d2367949: Generated new TSK 0
I20260925 13:21:21.606356 21120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:21.611354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 21033
2026-09-25T13:21:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:21 dist-test-slave-t8d9 krb5kdc[21132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:21 dist-test-slave-t8d9 krb5kdc[21132](info): set up 2 sockets
Sep 25 13:21:21 dist-test-slave-t8d9 krb5kdc[21132](info): commencing operation
krb5kdc: starting...
W20260925 13:21:22.569337 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:22 dist-test-slave-t8d9 krb5kdc[21132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:21:22.682339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:22.763257 21145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:22.763456 21145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:22.763477 21145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:22.764595 21145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:22.764636 21145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:22.764648 21145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:22.764659 21145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:22.764670 21145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:22.766151 21145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:22.766387 21145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:22.766592 21145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:22.771138 21150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:22.771234 21145 server_base.cc:1034] running on GCE node
W20260925 13:21:22.771160 21151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:22.771144 21153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:22.771548 21145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:22.771804 21145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:22.772969 21145 hybrid_clock.cc:648] HybridClock initialized: now 1790342482772949 us; error 34 us; skew 500 ppm
Sep 25 13:21:22 dist-test-slave-t8d9 krb5kdc[21132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:22.775362 21145 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:21:22.775759 21145 webserver.cc:460] Webserver started at http://127.0.0.1:42747/ using document root <none> and password file <none>
I20260925 13:21:22.775990 21145 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:22.776039 21145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:22.776165 21145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:22.777206 21145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "fd9c717820284dae9ce939da274e5e6d"
format_stamp: "Formatted at 2026-09-25 13:21:22 on dist-test-slave-t8d9"
server_key: "1a7f9a18ee3cca1e8e0ba666a82b9b52"
server_key_iv: "1d6ee20651affbc6583ba82a5145628b"
server_key_version: "encryptionkey@0"
I20260925 13:21:22.777566 21145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "fd9c717820284dae9ce939da274e5e6d"
format_stamp: "Formatted at 2026-09-25 13:21:22 on dist-test-slave-t8d9"
server_key: "1a7f9a18ee3cca1e8e0ba666a82b9b52"
server_key_iv: "1d6ee20651affbc6583ba82a5145628b"
server_key_version: "encryptionkey@0"
I20260925 13:21:22.778786 21145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:21:22.779523 21160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:22.779687 21145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:21:22.779769 21145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "fd9c717820284dae9ce939da274e5e6d"
format_stamp: "Formatted at 2026-09-25 13:21:22 on dist-test-slave-t8d9"
server_key: "1a7f9a18ee3cca1e8e0ba666a82b9b52"
server_key_iv: "1d6ee20651affbc6583ba82a5145628b"
server_key_version: "encryptionkey@0"
I20260925 13:21:22.779835 21145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:22.783054 21145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:22.784868 21145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:22.785001 21145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:22.788587 21145 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33629
I20260925 13:21:22.788638 21212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33629 every 8 connection(s)
I20260925 13:21:22.788911 21145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:22.789904 21213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:22.792147 21213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d: Bootstrap starting.
I20260925 13:21:22.792661 21213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:22.792914 21213 log.cc:824] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:22.793588 21213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d: No bootstrap required, opened a new log
I20260925 13:21:22.795187 21213 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9c717820284dae9ce939da274e5e6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33629 } }
I20260925 13:21:22.795331 21213 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:22.795356 21213 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9c717820284dae9ce939da274e5e6d, State: Initialized, Role: FOLLOWER
I20260925 13:21:22.795467 21213 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9c717820284dae9ce939da274e5e6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33629 } }
I20260925 13:21:22.795543 21213 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:22.795573 21213 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:22.795609 21213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:22.796118 21213 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9c717820284dae9ce939da274e5e6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33629 } }
I20260925 13:21:22.796204 21213 leader_election.cc:302] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd9c717820284dae9ce939da274e5e6d; no voters:
I20260925 13:21:22.796331 21213 leader_election.cc:288] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:22.796389 21216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:22.796583 21213 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:22.796568 21216 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [term 1 LEADER]: Becoming Leader. State: Replica: fd9c717820284dae9ce939da274e5e6d, State: Running, Role: LEADER
I20260925 13:21:22.796703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 21145
I20260925 13:21:22.796799 21216 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9c717820284dae9ce939da274e5e6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33629 } }
I20260925 13:21:22.796803 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:22.797024 392 external_mini_cluster.cc:1428] Setting key 3055b032c416e034a4218c4c8201b178
I20260925 13:21:22.797851 21218 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd9c717820284dae9ce939da274e5e6d. Latest consensus state: current_term: 1 leader_uuid: "fd9c717820284dae9ce939da274e5e6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9c717820284dae9ce939da274e5e6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33629 } } }
I20260925 13:21:22.797945 21218 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:22.798189 21217 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd9c717820284dae9ce939da274e5e6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9c717820284dae9ce939da274e5e6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33629 } } }
I20260925 13:21:22.798262 21217 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d [sys.catalog]: This master's current role is: LEADER
W20260925 13:21:22.798569 21233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:21:22.798657 21233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:21:22.798796 21234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:22.798961 21234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:22.799700 21226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:22.797462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59029 (local address 127.0.0.1:33629)
0925 13:21:22.798217 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:22.798227 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:22.798262 (+ 35us) server_negotiation.cc:407] Connection header received
0925 13:21:22.798384 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:22.798391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:22.798418 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:22.798555 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:22.799264 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:22.799268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:22.799268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:22.799273 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:22.799324 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:22.799335 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:22.799395 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:22.799490 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 13:21:22.799532 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":96,"threads_started":1}
I20260925 13:21:22.800563 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:22.801642 21234 catalog_manager.cc:1347] Generated new cluster ID: e7b98d34a0b84a8b822e844fb579f046
I20260925 13:21:22.801735 21234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:22.803292 21226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:22.802142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38405 (local address 127.0.0.1:33629)
0925 13:21:22.802180 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:22.802182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:22.802362 (+ 180us) server_negotiation.cc:407] Connection header received
0925 13:21:22.802419 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:22.802420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:22.802425 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:22.802484 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:22.803078 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:22.803079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:22.803080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:22.803084 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:22.803100 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:22.803105 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:22.803162 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:22.803210 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 13:21:22.803223 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:21:22.814182 21234 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:22.814353 21234 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:22.817998 21234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd9c717820284dae9ce939da274e5e6d: Generated new TSK 0
I20260925 13:21:22.818212 21234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:22.825555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 21145
2026-09-25T13:21:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:22 dist-test-slave-t8d9 krb5kdc[21244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:22 dist-test-slave-t8d9 krb5kdc[21244](info): set up 2 sockets
Sep 25 13:21:22 dist-test-slave-t8d9 krb5kdc[21244](info): commencing operation
krb5kdc: starting...
W20260925 13:21:23.769183 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:23 dist-test-slave-t8d9 krb5kdc[21244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:21:23.880903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:23.959005 21257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:23.959203 21257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:23.959232 21257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:23.960346 21257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:23.960386 21257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:23.960404 21257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:23.960418 21257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:23.960434 21257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:23.961910 21257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:23.962133 21257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:23.962327 21257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:23.966689 21265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:23.966704 21263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:23.966748 21262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:23.966867 21257 server_base.cc:1034] running on GCE node
I20260925 13:21:23.967154 21257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:23.967394 21257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:23.968554 21257 hybrid_clock.cc:648] HybridClock initialized: now 1790342483968533 us; error 31 us; skew 500 ppm
Sep 25 13:21:23 dist-test-slave-t8d9 krb5kdc[21244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:23.975848 21257 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:21:23.976240 21257 webserver.cc:460] Webserver started at http://127.0.0.1:40555/ using document root <none> and password file <none>
I20260925 13:21:23.976405 21257 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:23.976444 21257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:23.976529 21257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:23.977490 21257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "e9f4fcbd6491441d8b2e2dbb0f900e70"
format_stamp: "Formatted at 2026-09-25 13:21:23 on dist-test-slave-t8d9"
server_key: "ec246cbc05cb00b707d87fc0a43dedc9"
server_key_iv: "eeb0a77b4b6ff5f4df037723e75cbc0d"
server_key_version: "encryptionkey@0"
I20260925 13:21:23.977808 21257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "e9f4fcbd6491441d8b2e2dbb0f900e70"
format_stamp: "Formatted at 2026-09-25 13:21:23 on dist-test-slave-t8d9"
server_key: "ec246cbc05cb00b707d87fc0a43dedc9"
server_key_iv: "eeb0a77b4b6ff5f4df037723e75cbc0d"
server_key_version: "encryptionkey@0"
I20260925 13:21:23.979009 21257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:21:23.979656 21272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:23.979825 21257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:23.979911 21257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "e9f4fcbd6491441d8b2e2dbb0f900e70"
format_stamp: "Formatted at 2026-09-25 13:21:23 on dist-test-slave-t8d9"
server_key: "ec246cbc05cb00b707d87fc0a43dedc9"
server_key_iv: "eeb0a77b4b6ff5f4df037723e75cbc0d"
server_key_version: "encryptionkey@0"
I20260925 13:21:23.979962 21257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:23.998682 21257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:24.000509 21257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:24.000620 21257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:24.004232 21257 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46827
I20260925 13:21:24.004279 21324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46827 every 8 connection(s)
I20260925 13:21:24.004590 21257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:24.005347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 21257
I20260925 13:21:24.005463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:24.005492 21325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:24.005710 392 external_mini_cluster.cc:1428] Setting key c60e46962fe12a9d2df255ea8e17c7e3
I20260925 13:21:24.008216 21325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70: Bootstrap starting.
I20260925 13:21:24.008790 21325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:24.009032 21325 log.cc:824] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:24.009699 21325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70: No bootstrap required, opened a new log
Sep 25 13:21:24 dist-test-slave-t8d9 krb5kdc[21244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 13:21:24.011758 21325 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f4fcbd6491441d8b2e2dbb0f900e70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260925 13:21:24.011915 21325 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:24.011958 21325 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f4fcbd6491441d8b2e2dbb0f900e70, State: Initialized, Role: FOLLOWER
I20260925 13:21:24.012082 21325 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f4fcbd6491441d8b2e2dbb0f900e70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260925 13:21:24.012156 21325 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:24.012195 21325 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:24.012243 21325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:24.012830 21325 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f4fcbd6491441d8b2e2dbb0f900e70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260925 13:21:24.012938 21325 leader_election.cc:302] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9f4fcbd6491441d8b2e2dbb0f900e70; no voters:
I20260925 13:21:24.013092 21325 leader_election.cc:288] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:24.013262 21331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:24.013368 21325 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:24.013572 21331 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [term 1 LEADER]: Becoming Leader. State: Replica: e9f4fcbd6491441d8b2e2dbb0f900e70, State: Running, Role: LEADER
I20260925 13:21:24.013706 21331 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f4fcbd6491441d8b2e2dbb0f900e70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260925 13:21:24.014034 21330 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9f4fcbd6491441d8b2e2dbb0f900e70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f4fcbd6491441d8b2e2dbb0f900e70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } } }
I20260925 13:21:24.014103 21330 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:24.014251 21332 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f4fcbd6491441d8b2e2dbb0f900e70. Latest consensus state: current_term: 1 leader_uuid: "e9f4fcbd6491441d8b2e2dbb0f900e70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f4fcbd6491441d8b2e2dbb0f900e70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } } }
I20260925 13:21:24.014310 21332 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:24.014616 21339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:24.015161 21339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:24.016255 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:24.006232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.123:58921 (local address 127.0.0.1:46827)
0925 13:21:24.006413 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:24.006420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:24.006445 (+ 25us) server_negotiation.cc:407] Connection header received
0925 13:21:24.006868 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:24.006874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:24.006932 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:24.007054 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:24.007661 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:24.008580 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:24.009372 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:24.009823 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:24.009952 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:24.012325 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:24.012330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:24.012330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:24.012342 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:24.014320 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:24.014769 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:24.014771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:24.014772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:24.014866 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:24.015124 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:24.015125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:24.015125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:24.015412 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:24.015935 (+ 523us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:24.016000 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 13:21:24.016071 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260925 13:21:24.016809 21339 catalog_manager.cc:1347] Generated new cluster ID: 2535cebcc64b4e049c0188b60e5b1d25
I20260925 13:21:24.016858 21339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:24.017141 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:24.023911 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:24.018356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.123:40515 (local address 127.0.0.1:46827)
0925 13:21:24.018389 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:24.018390 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:24.018484 (+ 94us) server_negotiation.cc:407] Connection header received
0925 13:21:24.018532 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:24.018533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:24.018542 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:24.018584 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:24.019024 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:24.019743 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:24.020385 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:24.020604 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:24.020657 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:24.021737 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:24.021738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:24.021738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:24.021744 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:24.022930 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:24.023239 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:24.023240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:24.023240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:24.023288 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:24.023471 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:24.023472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:24.023472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:24.023571 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:24.023655 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:24.023765 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 13:21:24.023818 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:21:24.031006 21339 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:24.031477 21339 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:24.040705 21339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9f4fcbd6491441d8b2e2dbb0f900e70: Generated new TSK 0
I20260925 13:21:24.040882 21339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:24.050298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 21257
W20260925 13:21:24.053941 21242 connection.cc:535] client connection to 127.0.0.1:46827 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46827): Connection reset by peer (error 104)
2026-09-25T13:21:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:24 dist-test-slave-t8d9 krb5kdc[21356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:24 dist-test-slave-t8d9 krb5kdc[21356](info): set up 2 sockets
Sep 25 13:21:24 dist-test-slave-t8d9 krb5kdc[21356](info): commencing operation
krb5kdc: starting...
W20260925 13:21:24.989243 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:25 dist-test-slave-t8d9 krb5kdc[21356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:21:25.100536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:25.182145 21369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:25.182366 21369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:25.182390 21369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:25.183555 21369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:25.183604 21369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:25.183619 21369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:25.183645 21369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:25.183671 21369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:25.185209 21369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:25.185447 21369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:25.185711 21369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:25.190310 21375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:25.190412 21369 server_base.cc:1034] running on GCE node
W20260925 13:21:25.190317 21377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:25.190348 21374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:25.190737 21369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:25.191026 21369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:25.192169 21369 hybrid_clock.cc:648] HybridClock initialized: now 1790342485192153 us; error 31 us; skew 500 ppm
Sep 25 13:21:25 dist-test-slave-t8d9 krb5kdc[21356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:25.194537 21369 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:21:25.194933 21369 webserver.cc:460] Webserver started at http://127.0.0.1:44553/ using document root <none> and password file <none>
I20260925 13:21:25.195133 21369 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:25.195180 21369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:25.195302 21369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:25.196363 21369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "6eabce7b5429491294ab47168dc91e6e"
format_stamp: "Formatted at 2026-09-25 13:21:25 on dist-test-slave-t8d9"
server_key: "995a5d85f4234fdbb79255c5b6c67693"
server_key_iv: "383f8c5c963f63535766edce19352e48"
server_key_version: "encryptionkey@0"
I20260925 13:21:25.196684 21369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "6eabce7b5429491294ab47168dc91e6e"
format_stamp: "Formatted at 2026-09-25 13:21:25 on dist-test-slave-t8d9"
server_key: "995a5d85f4234fdbb79255c5b6c67693"
server_key_iv: "383f8c5c963f63535766edce19352e48"
server_key_version: "encryptionkey@0"
I20260925 13:21:25.197944 21369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 13:21:25.198575 21384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:25.198728 21369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:25.198819 21369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "6eabce7b5429491294ab47168dc91e6e"
format_stamp: "Formatted at 2026-09-25 13:21:25 on dist-test-slave-t8d9"
server_key: "995a5d85f4234fdbb79255c5b6c67693"
server_key_iv: "383f8c5c963f63535766edce19352e48"
server_key_version: "encryptionkey@0"
I20260925 13:21:25.198875 21369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:25.217077 21369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:25.219377 21369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:25.219521 21369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:25.223220 21369 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43725
I20260925 13:21:25.223263 21436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43725 every 8 connection(s)
I20260925 13:21:25.223587 21369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:25.224421 21437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:25.225412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 21369
I20260925 13:21:25.225561 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:25.225767 392 external_mini_cluster.cc:1428] Setting key b37077afde0965f19db87fef9cec5cb9
I20260925 13:21:25.227187 21437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e: Bootstrap starting.
I20260925 13:21:25.227808 21437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:25.228071 21437 log.cc:824] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:25.228691 21437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e: No bootstrap required, opened a new log
I20260925 13:21:25.230484 21437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eabce7b5429491294ab47168dc91e6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43725 } }
I20260925 13:21:25.230625 21437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:25.230657 21437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eabce7b5429491294ab47168dc91e6e, State: Initialized, Role: FOLLOWER
I20260925 13:21:25.230804 21437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eabce7b5429491294ab47168dc91e6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43725 } }
I20260925 13:21:25.230883 21437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:25.230914 21437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:25.230962 21437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:25.231124 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:25.226292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.123:60383 (local address 127.0.0.1:43725)
0925 13:21:25.226459 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:25.226469 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:25.226494 (+ 25us) server_negotiation.cc:407] Connection header received
0925 13:21:25.226973 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:25.226980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:25.227033 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:25.227152 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:25.227383 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:25.228386 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:25.229183 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:25.229656 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:25.229752 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:25.230386 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:25.230390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:25.230391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:25.230566 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:25.230598 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:25.230608 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:25.230677 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:25.230770 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 13:21:25.230883 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260925 13:21:25.231568 21437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eabce7b5429491294ab47168dc91e6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43725 } }
I20260925 13:21:25.231662 21437 leader_election.cc:302] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eabce7b5429491294ab47168dc91e6e; no voters:
I20260925 13:21:25.231822 21437 leader_election.cc:288] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:25.231871 21442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:25.231981 21442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [term 1 LEADER]: Becoming Leader. State: Replica: 6eabce7b5429491294ab47168dc91e6e, State: Running, Role: LEADER
I20260925 13:21:25.231992 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:25.232065 21437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:25.232106 21442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eabce7b5429491294ab47168dc91e6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43725 } }
I20260925 13:21:25.232403 21442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eabce7b5429491294ab47168dc91e6e. Latest consensus state: current_term: 1 leader_uuid: "6eabce7b5429491294ab47168dc91e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eabce7b5429491294ab47168dc91e6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43725 } } }
I20260925 13:21:25.232466 21442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:25.232599 21443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6eabce7b5429491294ab47168dc91e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eabce7b5429491294ab47168dc91e6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43725 } } }
I20260925 13:21:25.232645 21443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:25.232749 21448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:25.233229 21448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:25.234724 21448 catalog_manager.cc:1347] Generated new cluster ID: ca9a7ef4009a4adbbfa08f3e8d4bf2fb
I20260925 13:21:25.234777 21448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:25.236331 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:25.233608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.123:42425 (local address 127.0.0.1:43725)
0925 13:21:25.233640 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:25.233642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:25.233647 (+ 5us) server_negotiation.cc:407] Connection header received
0925 13:21:25.233664 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:25.233665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:25.233674 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:25.233716 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:25.233863 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:25.234508 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:25.235266 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:25.235471 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:25.235524 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:25.236115 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:25.236116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:25.236117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:25.236121 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:25.236136 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:25.236140 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:25.236186 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:25.236252 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 13:21:25.236260 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 13:21:25.254254 21448 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:25.254797 21448 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:25.257751 21448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6eabce7b5429491294ab47168dc91e6e: Generated new TSK 0
I20260925 13:21:25.257951 21448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:25.263551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 21369
W20260925 13:21:25.267223 21354 connection.cc:535] client connection to 127.0.0.1:43725 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43725): Connection reset by peer (error 104)
2026-09-25T13:21:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:25 dist-test-slave-t8d9 krb5kdc[21468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:25 dist-test-slave-t8d9 krb5kdc[21468](info): set up 2 sockets
Sep 25 13:21:25 dist-test-slave-t8d9 krb5kdc[21468](info): commencing operation
krb5kdc: starting...
W20260925 13:21:26.213256 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:26 dist-test-slave-t8d9 krb5kdc[21468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:21:26.326617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:26.404563 21481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:26.404772 21481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:26.404794 21481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:26.406000 21481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:26.406045 21481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:26.406059 21481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:26.406069 21481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:26.406081 21481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:26.407611 21481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:26.407847 21481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:26.408071 21481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:26.412655 21489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:26.412660 21487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:26.412667 21486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:26.413074 21481 server_base.cc:1034] running on GCE node
I20260925 13:21:26.413355 21481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:26.413702 21481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:26.414849 21481 hybrid_clock.cc:648] HybridClock initialized: now 1790342486414834 us; error 36 us; skew 500 ppm
Sep 25 13:21:26 dist-test-slave-t8d9 krb5kdc[21468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:26.417886 21481 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:21:26.418419 21481 webserver.cc:460] Webserver started at http://127.0.0.1:32907/ using document root <none> and password file <none>
I20260925 13:21:26.418696 21481 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:26.418799 21481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:26.418943 21481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:26.420238 21481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "7e255c5a8f4b4e3d8f05bf6f1a641094"
format_stamp: "Formatted at 2026-09-25 13:21:26 on dist-test-slave-t8d9"
server_key: "2a81fb74025ce22e8ca22b9049a3844d"
server_key_iv: "4d4f1dcfb1f4b07777b76bb7f84bb3f3"
server_key_version: "encryptionkey@0"
I20260925 13:21:26.420619 21481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e255c5a8f4b4e3d8f05bf6f1a641094"
format_stamp: "Formatted at 2026-09-25 13:21:26 on dist-test-slave-t8d9"
server_key: "2a81fb74025ce22e8ca22b9049a3844d"
server_key_iv: "4d4f1dcfb1f4b07777b76bb7f84bb3f3"
server_key_version: "encryptionkey@0"
I20260925 13:21:26.422293 21481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:21:26.423017 21496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:26.423169 21481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:26.423261 21481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "7e255c5a8f4b4e3d8f05bf6f1a641094"
format_stamp: "Formatted at 2026-09-25 13:21:26 on dist-test-slave-t8d9"
server_key: "2a81fb74025ce22e8ca22b9049a3844d"
server_key_iv: "4d4f1dcfb1f4b07777b76bb7f84bb3f3"
server_key_version: "encryptionkey@0"
I20260925 13:21:26.423328 21481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:26.426587 21481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:26.428503 21481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:26.428644 21481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:26.432636 21481 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33853
I20260925 13:21:26.432667 21548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33853 every 8 connection(s)
I20260925 13:21:26.432986 21481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:26.433945 21549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:26.436178 21549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094: Bootstrap starting.
I20260925 13:21:26.436736 21549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:26.436995 21549 log.cc:824] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:26.437695 21549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094: No bootstrap required, opened a new log
I20260925 13:21:26.439286 21549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e255c5a8f4b4e3d8f05bf6f1a641094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } }
I20260925 13:21:26.439433 21549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:26.439467 21549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e255c5a8f4b4e3d8f05bf6f1a641094, State: Initialized, Role: FOLLOWER
I20260925 13:21:26.439608 21549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e255c5a8f4b4e3d8f05bf6f1a641094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } }
I20260925 13:21:26.439669 21549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:26.439692 21549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:26.439735 21549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:26.440265 21549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e255c5a8f4b4e3d8f05bf6f1a641094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } }
I20260925 13:21:26.440356 21549 leader_election.cc:302] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e255c5a8f4b4e3d8f05bf6f1a641094; no voters:
I20260925 13:21:26.440526 21549 leader_election.cc:288] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:26.440569 21552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:26.440716 21552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [term 1 LEADER]: Becoming Leader. State: Replica: 7e255c5a8f4b4e3d8f05bf6f1a641094, State: Running, Role: LEADER
I20260925 13:21:26.440778 21549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:26.440805 21552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e255c5a8f4b4e3d8f05bf6f1a641094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } }
I20260925 13:21:26.441093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 21481
I20260925 13:21:26.441205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:26.441339 21553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e255c5a8f4b4e3d8f05bf6f1a641094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e255c5a8f4b4e3d8f05bf6f1a641094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } } }
I20260925 13:21:26.441390 392 external_mini_cluster.cc:1428] Setting key 00abd15e2876c804a68801ba6389ae67
I20260925 13:21:26.441407 21553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:26.441588 21554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e255c5a8f4b4e3d8f05bf6f1a641094. Latest consensus state: current_term: 1 leader_uuid: "7e255c5a8f4b4e3d8f05bf6f1a641094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e255c5a8f4b4e3d8f05bf6f1a641094" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } } }
I20260925 13:21:26.441646 21554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094 [sys.catalog]: This master's current role is: LEADER
W20260925 13:21:26.442821 21570 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:21:26.443452 21570 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:21:26.443740 21561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:26.444046 21561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:26.444047 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:26.441847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.123:60095 (local address 127.0.0.1:33853)
0925 13:21:26.441999 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:26.442008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:26.442530 (+ 522us) server_negotiation.cc:407] Connection header received
0925 13:21:26.442725 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:26.442734 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:26.442771 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:26.442897 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:26.443610 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:26.443613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:26.443613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:26.443617 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:26.443654 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:26.443664 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:26.443715 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:26.443812 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 13:21:26.443859 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260925 13:21:26.445048 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:26.445498 21561 catalog_manager.cc:1347] Generated new cluster ID: e75d4525257044f2894c2e6de1264186
I20260925 13:21:26.445562 21561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:26.449880 21561 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:26.450043 21561 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:26.454047 21561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e255c5a8f4b4e3d8f05bf6f1a641094: Generated new TSK 0
I20260925 13:21:26.454238 21561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:26.454913 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:26.449734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.123:53517 (local address 127.0.0.1:33853)
0925 13:21:26.449815 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:26.449817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:26.453793 (+ 3976us) server_negotiation.cc:407] Connection header received
0925 13:21:26.453887 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:26.453888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:26.453894 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:26.453949 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:26.454697 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:26.454698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:26.454698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:26.454702 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:26.454718 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:26.454722 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:26.454771 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:26.454818 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 13:21:26.454833 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 13:21:26.456207 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 21481
2026-09-25T13:21:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1192 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:26 dist-test-slave-t8d9 krb5kdc[21580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:26 dist-test-slave-t8d9 krb5kdc[21580](info): set up 2 sockets
Sep 25 13:21:26 dist-test-slave-t8d9 krb5kdc[21580](info): commencing operation
krb5kdc: starting...
W20260925 13:21:27.401371 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:27 dist-test-slave-t8d9 krb5kdc[21580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:21:27.511778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36517
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:27.590866 21593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:27.591084 21593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:27.591106 21593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:27.592216 21593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:27.592252 21593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:27.592264 21593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:27.592275 21593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:27.592290 21593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:27.593732 21593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36517
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:27.593935 21593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:27.594103 21593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:27.598774 21598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:27.598888 21593 server_base.cc:1034] running on GCE node
W20260925 13:21:27.598757 21601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:27.599035 21599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:27.599278 21593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:27.599542 21593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:27.600689 21593 hybrid_clock.cc:648] HybridClock initialized: now 1790342487600668 us; error 32 us; skew 500 ppm
Sep 25 13:21:27 dist-test-slave-t8d9 krb5kdc[21580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342487, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:27.603045 21593 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:21:27.603473 21593 webserver.cc:460] Webserver started at http://127.0.0.1:45745/ using document root <none> and password file <none>
I20260925 13:21:27.603681 21593 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:27.603722 21593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:27.603848 21593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:27.604908 21593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "5e7b9589be254f258e5c53eeb6784524"
format_stamp: "Formatted at 2026-09-25 13:21:27 on dist-test-slave-t8d9"
server_key: "b19f0148358eb3fc5bede524d5ed9fc3"
server_key_iv: "109c44c4e5f143385beb57f58d4f04f3"
server_key_version: "encryptionkey@0"
I20260925 13:21:27.605240 21593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e7b9589be254f258e5c53eeb6784524"
format_stamp: "Formatted at 2026-09-25 13:21:27 on dist-test-slave-t8d9"
server_key: "b19f0148358eb3fc5bede524d5ed9fc3"
server_key_iv: "109c44c4e5f143385beb57f58d4f04f3"
server_key_version: "encryptionkey@0"
I20260925 13:21:27.606604 21593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 13:21:27.607322 21608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:27.607530 21593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:21:27.607612 21593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "5e7b9589be254f258e5c53eeb6784524"
format_stamp: "Formatted at 2026-09-25 13:21:27 on dist-test-slave-t8d9"
server_key: "b19f0148358eb3fc5bede524d5ed9fc3"
server_key_iv: "109c44c4e5f143385beb57f58d4f04f3"
server_key_version: "encryptionkey@0"
I20260925 13:21:27.607671 21593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:27.619323 21593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:27.621302 21593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:27.621412 21593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:27.624975 21593 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36517
I20260925 13:21:27.625032 21660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36517 every 8 connection(s)
I20260925 13:21:27.625321 21593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:27.626044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 21593
I20260925 13:21:27.626175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:27.626322 21661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:27.626379 392 external_mini_cluster.cc:1428] Setting key 9bb52b621fa499d671c7cf0effc7b5e9
I20260925 13:21:27.629010 21661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524: Bootstrap starting.
I20260925 13:21:27.629576 21661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:27.629827 21661 log.cc:824] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:27.630420 21661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524: No bootstrap required, opened a new log
I20260925 13:21:27.632275 21661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7b9589be254f258e5c53eeb6784524" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
Sep 25 13:21:27 dist-test-slave-t8d9 krb5kdc[21580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 13:21:27.632409 21661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:27.632444 21661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7b9589be254f258e5c53eeb6784524, State: Initialized, Role: FOLLOWER
I20260925 13:21:27.632567 21661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7b9589be254f258e5c53eeb6784524" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
I20260925 13:21:27.632627 21661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:27.632648 21661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:27.632692 21661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:27.633211 21661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7b9589be254f258e5c53eeb6784524" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
I20260925 13:21:27.633315 21661 leader_election.cc:302] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e7b9589be254f258e5c53eeb6784524; no voters:
I20260925 13:21:27.633473 21661 leader_election.cc:288] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:27.633500 21666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:27.633623 21666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [term 1 LEADER]: Becoming Leader. State: Replica: 5e7b9589be254f258e5c53eeb6784524, State: Running, Role: LEADER
I20260925 13:21:27.633751 21661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:27.633931 21666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7b9589be254f258e5c53eeb6784524" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
I20260925 13:21:27.635331 21667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e7b9589be254f258e5c53eeb6784524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7b9589be254f258e5c53eeb6784524" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } } }
I20260925 13:21:27.635540 21667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [sys.catalog]: This master's current role is: LEADER
W20260925 13:21:27.635635 21681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:21:27.635670 21681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:21:27.635732 21683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:27.635766 21668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e7b9589be254f258e5c53eeb6784524. Latest consensus state: current_term: 1 leader_uuid: "5e7b9589be254f258e5c53eeb6784524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7b9589be254f258e5c53eeb6784524" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } } }
I20260925 13:21:27.635820 21668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:27.635908 21683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:27.636593 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:27.626856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.123:53563 (local address 127.0.0.1:36517)
0925 13:21:27.627041 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:27.627048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:27.627071 (+ 23us) server_negotiation.cc:407] Connection header received
0925 13:21:27.627682 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:27.627688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:27.627739 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:27.627909 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:27.628458 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:27.629383 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:27.630202 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:27.630615 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:27.630707 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:27.632902 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:27.632906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:27.632906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:27.632915 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:27.634679 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:27.635071 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:27.635074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:27.635074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:27.635174 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:27.635390 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:27.635391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:27.635392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:27.635676 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:27.635820 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:27.636302 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 13:21:27.636403 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260925 13:21:27.637445 21683 catalog_manager.cc:1347] Generated new cluster ID: 1ddf34fa7a504f28882f0dcc25d97724
I20260925 13:21:27.637501 21683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:27.637687 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:27.643662 21683 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:27.644416 21683 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:27.647818 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:27.638935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.123:54439 (local address 127.0.0.1:36517)
0925 13:21:27.638966 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:27.638967 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:27.641766 (+ 2799us) server_negotiation.cc:407] Connection header received
0925 13:21:27.641832 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:27.641832 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:27.641845 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:27.641894 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:27.642427 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:27.642921 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:27.643847 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:27.644081 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:27.644136 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:27.645358 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:27.645360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:27.645360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:27.645367 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:27.646722 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:27.647079 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:27.647080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:27.647080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:27.647138 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:27.647327 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:27.647328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:27.647328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:27.647447 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:27.647537 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:27.647671 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 13:21:27.647731 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 13:21:27.648423 21683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e7b9589be254f258e5c53eeb6784524: Generated new TSK 0
I20260925 13:21:27.648643 21683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:27.652846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 21593
W20260925 13:21:27.656839 21578 connection.cc:535] client connection to 127.0.0.1:36517 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36517): Connection reset by peer (error 104)
2026-09-25T13:21:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1196 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:27 dist-test-slave-t8d9 krb5kdc[21692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:27 dist-test-slave-t8d9 krb5kdc[21692](info): set up 2 sockets
Sep 25 13:21:27 dist-test-slave-t8d9 krb5kdc[21692](info): commencing operation
krb5kdc: starting...
W20260925 13:21:28.601338 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:28 dist-test-slave-t8d9 krb5kdc[21692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:21:28.714178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:28.793274 21705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:28.793478 21705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:28.793499 21705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:28.794648 21705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:28.794695 21705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:28.794708 21705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:28.794719 21705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:28.794730 21705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:28.796168 21705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44157
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:28.796360 21705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:28.796538 21705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:28.801108 21710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:28.801311 21713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:28.801206 21711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:28.801352 21705 server_base.cc:1034] running on GCE node
I20260925 13:21:28.801664 21705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:28.801921 21705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:28.803069 21705 hybrid_clock.cc:648] HybridClock initialized: now 1790342488803059 us; error 27 us; skew 500 ppm
Sep 25 13:21:28 dist-test-slave-t8d9 krb5kdc[21692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:28.805424 21705 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:21:28.805905 21705 webserver.cc:460] Webserver started at http://127.0.0.1:36645/ using document root <none> and password file <none>
I20260925 13:21:28.806134 21705 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:28.806181 21705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:28.806301 21705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:28.807337 21705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "93f8f73d3c364b4db3ea0ab88c0d8d19"
format_stamp: "Formatted at 2026-09-25 13:21:28 on dist-test-slave-t8d9"
server_key: "15e01fb9202ea2228370262f7b79ab16"
server_key_iv: "2c3a4bdf33a7d66678c66151ce41d259"
server_key_version: "encryptionkey@0"
I20260925 13:21:28.807659 21705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "93f8f73d3c364b4db3ea0ab88c0d8d19"
format_stamp: "Formatted at 2026-09-25 13:21:28 on dist-test-slave-t8d9"
server_key: "15e01fb9202ea2228370262f7b79ab16"
server_key_iv: "2c3a4bdf33a7d66678c66151ce41d259"
server_key_version: "encryptionkey@0"
I20260925 13:21:28.808897 21705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:21:28.809643 21720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:28.809787 21705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:28.809868 21705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "93f8f73d3c364b4db3ea0ab88c0d8d19"
format_stamp: "Formatted at 2026-09-25 13:21:28 on dist-test-slave-t8d9"
server_key: "15e01fb9202ea2228370262f7b79ab16"
server_key_iv: "2c3a4bdf33a7d66678c66151ce41d259"
server_key_version: "encryptionkey@0"
I20260925 13:21:28.809929 21705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:28.819895 21705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:28.822003 21705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:28.822148 21705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:28.825889 21705 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44157
I20260925 13:21:28.825975 21772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44157 every 8 connection(s)
I20260925 13:21:28.826277 21705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:28.827257 21773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:28.828583 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 21705
I20260925 13:21:28.828675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:28.828852 392 external_mini_cluster.cc:1428] Setting key 3fca35930a048808a95a0c055153813c
I20260925 13:21:28.829777 21773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19: Bootstrap starting.
I20260925 13:21:28.830448 21773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:28.830755 21773 log.cc:824] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:28.831544 21773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19: No bootstrap required, opened a new log
I20260925 13:21:28.833621 21773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f8f73d3c364b4db3ea0ab88c0d8d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44157 } }
I20260925 13:21:28.833779 21773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:28.833812 21773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f8f73d3c364b4db3ea0ab88c0d8d19, State: Initialized, Role: FOLLOWER
I20260925 13:21:28.833932 21773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f8f73d3c364b4db3ea0ab88c0d8d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44157 } }
I20260925 13:21:28.834002 21773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:28.834030 21773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:28.834074 21773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:28.834800 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:28.829375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.123:50485 (local address 127.0.0.1:44157)
0925 13:21:28.829569 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:28.829579 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:28.829616 (+ 37us) server_negotiation.cc:407] Connection header received
0925 13:21:28.830138 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:28.830148 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:28.830216 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:28.830365 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:28.830533 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:28.831964 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:28.832748 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:28.833406 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:28.833516 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:28.834148 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:28.834152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:28.834152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:28.834350 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:28.834387 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:28.834403 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:28.834492 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:28.834564 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 13:21:28.834593 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260925 13:21:28.834792 21773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f8f73d3c364b4db3ea0ab88c0d8d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44157 } }
I20260925 13:21:28.834921 21773 leader_election.cc:302] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93f8f73d3c364b4db3ea0ab88c0d8d19; no voters:
I20260925 13:21:28.835085 21773 leader_election.cc:288] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:28.835134 21778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:28.835271 21778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [term 1 LEADER]: Becoming Leader. State: Replica: 93f8f73d3c364b4db3ea0ab88c0d8d19, State: Running, Role: LEADER
I20260925 13:21:28.835363 21773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:28.835354 21778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f8f73d3c364b4db3ea0ab88c0d8d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44157 } }
I20260925 13:21:28.835777 21778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93f8f73d3c364b4db3ea0ab88c0d8d19. Latest consensus state: current_term: 1 leader_uuid: "93f8f73d3c364b4db3ea0ab88c0d8d19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f8f73d3c364b4db3ea0ab88c0d8d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44157 } } }
I20260925 13:21:28.835837 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:28.835789 21780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93f8f73d3c364b4db3ea0ab88c0d8d19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f8f73d3c364b4db3ea0ab88c0d8d19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44157 } } }
I20260925 13:21:28.835932 21780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:28.836110 21778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:28.836115 21786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:28.836671 21786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:28.838186 21786 catalog_manager.cc:1347] Generated new cluster ID: 7eb639febde948fab2ba5900fc51fa4c
I20260925 13:21:28.838236 21786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:28.840499 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:28.837831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.123:59289 (local address 127.0.0.1:44157)
0925 13:21:28.837864 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:28.837865 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:28.838036 (+ 171us) server_negotiation.cc:407] Connection header received
0925 13:21:28.838082 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:28.838083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:28.838093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:28.838136 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:28.838288 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:28.838745 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:28.839406 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:28.839619 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:28.839672 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:28.840277 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:28.840279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:28.840279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:28.840284 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:28.840298 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:28.840302 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:28.840345 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:28.840419 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 13:21:28.840427 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 13:21:28.846828 21786 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:28.847302 21786 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:28.854748 21786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93f8f73d3c364b4db3ea0ab88c0d8d19: Generated new TSK 0
I20260925 13:21:28.854905 21786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:28.862697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 21705
W20260925 13:21:28.866340 21690 connection.cc:535] client connection to 127.0.0.1:44157 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44157): Connection reset by peer (error 104)
2026-09-25T13:21:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:28 dist-test-slave-t8d9 krb5kdc[21804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:28 dist-test-slave-t8d9 krb5kdc[21804](info): set up 2 sockets
Sep 25 13:21:28 dist-test-slave-t8d9 krb5kdc[21804](info): commencing operation
krb5kdc: starting...
W20260925 13:21:29.810302 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:29 dist-test-slave-t8d9 krb5kdc[21804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 13:21:29.932648 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:30.012816 21817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:30.013049 21817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:30.013073 21817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:30.014322 21817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:30.014376 21817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:30.014389 21817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:30.014401 21817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:30.014411 21817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:30.015976 21817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45359
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:30.016228 21817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:30.016460 21817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:30.021272 21825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:30.021297 21823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:30.021308 21822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:30.021487 21817 server_base.cc:1034] running on GCE node
I20260925 13:21:30.021765 21817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:30.022068 21817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:30.023214 21817 hybrid_clock.cc:648] HybridClock initialized: now 1790342490023202 us; error 30 us; skew 500 ppm
Sep 25 13:21:30 dist-test-slave-t8d9 krb5kdc[21804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:30.025686 21817 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 13:21:30.026134 21817 webserver.cc:460] Webserver started at http://127.0.0.1:37813/ using document root <none> and password file <none>
I20260925 13:21:30.026316 21817 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:30.026357 21817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:30.026450 21817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:30.027457 21817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "6e870e0bcf464a32af1f9b71f99452f7"
format_stamp: "Formatted at 2026-09-25 13:21:30 on dist-test-slave-t8d9"
server_key: "1f7d3140cf7d8f8debc75d4c9d217f99"
server_key_iv: "5677eb8daa3f77a8594ad77855967695"
server_key_version: "encryptionkey@0"
I20260925 13:21:30.027776 21817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e870e0bcf464a32af1f9b71f99452f7"
format_stamp: "Formatted at 2026-09-25 13:21:30 on dist-test-slave-t8d9"
server_key: "1f7d3140cf7d8f8debc75d4c9d217f99"
server_key_iv: "5677eb8daa3f77a8594ad77855967695"
server_key_version: "encryptionkey@0"
I20260925 13:21:30.028945 21817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:21:30.029657 21832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:30.029832 21817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 13:21:30.029920 21817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "6e870e0bcf464a32af1f9b71f99452f7"
format_stamp: "Formatted at 2026-09-25 13:21:30 on dist-test-slave-t8d9"
server_key: "1f7d3140cf7d8f8debc75d4c9d217f99"
server_key_iv: "5677eb8daa3f77a8594ad77855967695"
server_key_version: "encryptionkey@0"
I20260925 13:21:30.029978 21817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:30.033288 21817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:30.035477 21817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:30.035593 21817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:30.039356 21817 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45359
I20260925 13:21:30.039384 21884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45359 every 8 connection(s)
I20260925 13:21:30.039718 21817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:30.040704 21885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:30.043205 21885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7: Bootstrap starting.
I20260925 13:21:30.043803 21885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:30.044076 21885 log.cc:824] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:30.044857 21885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7: No bootstrap required, opened a new log
I20260925 13:21:30.046650 21885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e870e0bcf464a32af1f9b71f99452f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45359 } }
I20260925 13:21:30.046808 21885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:30.046839 21885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e870e0bcf464a32af1f9b71f99452f7, State: Initialized, Role: FOLLOWER
I20260925 13:21:30.046972 21885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e870e0bcf464a32af1f9b71f99452f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45359 } }
I20260925 13:21:30.047036 21885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:30.047067 21885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:30.047114 21885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:30.047204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 21817
I20260925 13:21:30.047305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:30.047482 392 external_mini_cluster.cc:1428] Setting key 35571b6ae557a5a7c1ed7766b70b55b3
I20260925 13:21:30.047827 21885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e870e0bcf464a32af1f9b71f99452f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45359 } }
I20260925 13:21:30.047955 21885 leader_election.cc:302] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e870e0bcf464a32af1f9b71f99452f7; no voters:
I20260925 13:21:30.048189 21885 leader_election.cc:288] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:30.048216 21890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:30.048422 21890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [term 1 LEADER]: Becoming Leader. State: Replica: 6e870e0bcf464a32af1f9b71f99452f7, State: Running, Role: LEADER
I20260925 13:21:30.048557 21885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:30.048554 21890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e870e0bcf464a32af1f9b71f99452f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45359 } }
I20260925 13:21:30.049021 21890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e870e0bcf464a32af1f9b71f99452f7. Latest consensus state: current_term: 1 leader_uuid: "6e870e0bcf464a32af1f9b71f99452f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e870e0bcf464a32af1f9b71f99452f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45359 } } }
I20260925 13:21:30.049028 21891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e870e0bcf464a32af1f9b71f99452f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e870e0bcf464a32af1f9b71f99452f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45359 } } }
I20260925 13:21:30.049131 21890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:30.049162 21891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:30.049346 21894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:30.049983 21889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:30.047995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.123:56119 (local address 127.0.0.1:45359)
0925 13:21:30.048200 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:30.048211 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:30.048296 (+ 85us) server_negotiation.cc:407] Connection header received
0925 13:21:30.048429 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:30.048436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:30.048468 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:30.048614 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:30.049507 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:30.049511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:30.049511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:30.049516 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:30.049585 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:30.049602 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:30.049659 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:30.049720 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 13:21:30.049765 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":98,"threads_started":1}
I20260925 13:21:30.050055 21894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:30.050995 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:30.052002 21894 catalog_manager.cc:1347] Generated new cluster ID: 2b36986184de4b4b8ee30f3e2f47f462
I20260925 13:21:30.052058 21894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:30.053419 21889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:30.052495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.123:52109 (local address 127.0.0.1:45359)
0925 13:21:30.052544 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:30.052546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:30.052587 (+ 41us) server_negotiation.cc:407] Connection header received
0925 13:21:30.052647 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:30.052648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:30.052672 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:30.052720 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:30.053240 (+ 520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:30.053241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:30.053241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:30.053245 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:30.053258 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:30.053262 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:30.053306 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:30.053344 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 13:21:30.053358 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 13:21:30.062979 21894 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:30.063176 21894 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:30.071616 21894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e870e0bcf464a32af1f9b71f99452f7: Generated new TSK 0
I20260925 13:21:30.071889 21894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:30.079051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 21817
2026-09-25T13:21:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:30 dist-test-slave-t8d9 krb5kdc[21916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:30 dist-test-slave-t8d9 krb5kdc[21916](info): set up 2 sockets
Sep 25 13:21:30 dist-test-slave-t8d9 krb5kdc[21916](info): commencing operation
krb5kdc: starting...
W20260925 13:21:31.041455 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:31 dist-test-slave-t8d9 krb5kdc[21916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:21:31.160252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39521
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:31.241290 21929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:31.241588 21929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:31.241627 21929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:31.242866 21929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:31.242914 21929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:31.242933 21929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:31.242949 21929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:31.242964 21929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:31.244511 21929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39521
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:31.244787 21929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:31.245007 21929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:31.249744 21934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:31.249785 21929 server_base.cc:1034] running on GCE node
W20260925 13:21:31.249776 21935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:31.249752 21937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:31.250226 21929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:31.250520 21929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:31.251663 21929 hybrid_clock.cc:648] HybridClock initialized: now 1790342491251647 us; error 29 us; skew 500 ppm
Sep 25 13:21:31 dist-test-slave-t8d9 krb5kdc[21916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:31.254207 21929 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:21:31.254724 21929 webserver.cc:460] Webserver started at http://127.0.98.62:36779/ using document root <none> and password file <none>
I20260925 13:21:31.254948 21929 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:31.254999 21929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:31.255134 21929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:31.256282 21929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "0d2ed291fbe54ce5a2a444f20c1dd914"
format_stamp: "Formatted at 2026-09-25 13:21:31 on dist-test-slave-t8d9"
server_key: "8eff36b62fa8d742971697a93bbe7f64"
server_key_iv: "7bddf3d48e1633a80f868e6bc832c642"
server_key_version: "encryptionkey@0"
I20260925 13:21:31.256630 21929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d2ed291fbe54ce5a2a444f20c1dd914"
format_stamp: "Formatted at 2026-09-25 13:21:31 on dist-test-slave-t8d9"
server_key: "8eff36b62fa8d742971697a93bbe7f64"
server_key_iv: "7bddf3d48e1633a80f868e6bc832c642"
server_key_version: "encryptionkey@0"
I20260925 13:21:31.257927 21929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:21:31.258671 21944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:31.258826 21929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:21:31.258909 21929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "0d2ed291fbe54ce5a2a444f20c1dd914"
format_stamp: "Formatted at 2026-09-25 13:21:31 on dist-test-slave-t8d9"
server_key: "8eff36b62fa8d742971697a93bbe7f64"
server_key_iv: "7bddf3d48e1633a80f868e6bc832c642"
server_key_version: "encryptionkey@0"
I20260925 13:21:31.258970 21929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:31.270990 21929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:31.273025 21929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:31.273167 21929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:31.277087 21929 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39521
I20260925 13:21:31.277133 21996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39521 every 8 connection(s)
I20260925 13:21:31.277472 21929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:31.278400 21997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:31.280704 21997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914: Bootstrap starting.
I20260925 13:21:31.281316 21997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:31.281635 21997 log.cc:824] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:31.282372 21997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914: No bootstrap required, opened a new log
I20260925 13:21:31.284099 21997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2ed291fbe54ce5a2a444f20c1dd914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39521 } }
I20260925 13:21:31.284260 21997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:31.284298 21997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2ed291fbe54ce5a2a444f20c1dd914, State: Initialized, Role: FOLLOWER
I20260925 13:21:31.284458 21997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2ed291fbe54ce5a2a444f20c1dd914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39521 } }
I20260925 13:21:31.284528 21997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:31.284564 21997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:31.284617 21997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:31.285065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 21929
I20260925 13:21:31.285179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:31.285387 392 external_mini_cluster.cc:1428] Setting key a4d51c9c0582fd68bd3cbd831194554e
I20260925 13:21:31.285384 21997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2ed291fbe54ce5a2a444f20c1dd914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39521 } }
I20260925 13:21:31.285492 21997 leader_election.cc:302] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d2ed291fbe54ce5a2a444f20c1dd914; no voters:
I20260925 13:21:31.285750 21997 leader_election.cc:288] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:31.285805 22000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:31.286005 22000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [term 1 LEADER]: Becoming Leader. State: Replica: 0d2ed291fbe54ce5a2a444f20c1dd914, State: Running, Role: LEADER
I20260925 13:21:31.286098 21997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:31.286168 22000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2ed291fbe54ce5a2a444f20c1dd914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39521 } }
I20260925 13:21:31.286716 22002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d2ed291fbe54ce5a2a444f20c1dd914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2ed291fbe54ce5a2a444f20c1dd914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39521 } } }
I20260925 13:21:31.286813 22002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:31.287071 22003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d2ed291fbe54ce5a2a444f20c1dd914. Latest consensus state: current_term: 1 leader_uuid: "0d2ed291fbe54ce5a2a444f20c1dd914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2ed291fbe54ce5a2a444f20c1dd914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39521 } } }
I20260925 13:21:31.287204 22003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:31.287241 22011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:31.287789 22011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:31.289448 22011 catalog_manager.cc:1347] Generated new cluster ID: c7152c7eea694bffbdc947ca8a75b364
I20260925 13:21:31.289496 22011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:21:31 dist-test-slave-t8d9 krb5kdc[21916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:31.297799 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:31.285943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41149 (local address 127.0.98.62:39521)
0925 13:21:31.286405 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:31.286414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:31.286466 (+ 52us) server_negotiation.cc:407] Connection header received
0925 13:21:31.286601 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:31.286608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:31.286667 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:31.287593 (+ 926us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:31.287693 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:31.289148 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:31.290221 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:31.290843 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:31.290953 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:31.293877 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:31.293882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:31.293882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:31.293897 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:31.296019 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:31.296441 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:31.296444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:31.296444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:31.296593 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:31.296804 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:31.296805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:31.296805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:31.297142 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:31.297335 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:31.297439 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 13:21:31.297578 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":65,"threads_started":1}
I20260925 13:21:31.298789 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:31.304188 22011 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:31.304996 22011 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:31.309926 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:31.300221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45507 (local address 127.0.98.62:39521)
0925 13:21:31.300276 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:31.300277 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:31.304073 (+ 3796us) server_negotiation.cc:407] Connection header received
0925 13:21:31.304153 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:31.304154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:31.304166 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:31.304229 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:31.304855 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:31.305368 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:31.306104 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:31.306302 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:31.306357 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:31.307534 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:31.307535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:31.307536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:31.307544 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:31.308807 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:31.309165 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:31.309166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:31.309166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:31.309216 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:31.309406 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:31.309407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:31.309407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:31.309514 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:31.309667 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:31.309772 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 13:21:31.309834 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:21:31.322026 22011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d2ed291fbe54ce5a2a444f20c1dd914: Generated new TSK 0
I20260925 13:21:31.322271 22011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:31.328109 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 21929
W20260925 13:21:31.331816 21914 connection.cc:535] client connection to 127.0.98.62:39521 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39521): Connection reset by peer (error 104)
2026-09-25T13:21:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1248 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:31 dist-test-slave-t8d9 krb5kdc[22028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:31 dist-test-slave-t8d9 krb5kdc[22028](info): set up 2 sockets
Sep 25 13:21:31 dist-test-slave-t8d9 krb5kdc[22028](info): commencing operation
krb5kdc: starting...
W20260925 13:21:32.273391 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:32 dist-test-slave-t8d9 krb5kdc[22028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:21:32.386453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33217
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:32.466851 22041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:32.467051 22041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:32.467079 22041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:32.468218 22041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:32.468258 22041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:32.468271 22041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:32.468281 22041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:32.468303 22041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:32.469841 22041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33217
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:32.470060 22041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:32.470278 22041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:32.474761 22046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:32.474764 22047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:32.474921 22041 server_base.cc:1034] running on GCE node
W20260925 13:21:32.474813 22049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:32.475224 22041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:32.475490 22041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:32.476627 22041 hybrid_clock.cc:648] HybridClock initialized: now 1790342492476611 us; error 38 us; skew 500 ppm
Sep 25 13:21:32 dist-test-slave-t8d9 krb5kdc[22028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:32.478929 22041 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:21:32.479359 22041 webserver.cc:460] Webserver started at http://127.0.98.62:35195/ using document root <none> and password file <none>
I20260925 13:21:32.479562 22041 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:32.479612 22041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:32.479733 22041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:32.480742 22041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "f6e58d126bc04e409f4332a2361f4622"
format_stamp: "Formatted at 2026-09-25 13:21:32 on dist-test-slave-t8d9"
server_key: "cf75281d1c24223239c7fd88ac28dd9a"
server_key_iv: "3a65633dc15aa581c8b50c041dad1500"
server_key_version: "encryptionkey@0"
I20260925 13:21:32.481050 22041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6e58d126bc04e409f4332a2361f4622"
format_stamp: "Formatted at 2026-09-25 13:21:32 on dist-test-slave-t8d9"
server_key: "cf75281d1c24223239c7fd88ac28dd9a"
server_key_iv: "3a65633dc15aa581c8b50c041dad1500"
server_key_version: "encryptionkey@0"
I20260925 13:21:32.482263 22041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 13:21:32.482884 22056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:32.483044 22041 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 13:21:32.483131 22041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "f6e58d126bc04e409f4332a2361f4622"
format_stamp: "Formatted at 2026-09-25 13:21:32 on dist-test-slave-t8d9"
server_key: "cf75281d1c24223239c7fd88ac28dd9a"
server_key_iv: "3a65633dc15aa581c8b50c041dad1500"
server_key_version: "encryptionkey@0"
I20260925 13:21:32.483189 22041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:32.499400 22041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:32.501222 22041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:32.501355 22041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:32.504972 22041 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33217
I20260925 13:21:32.505017 22108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33217 every 8 connection(s)
I20260925 13:21:32.505340 22041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:32.506183 22109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:32.508450 22109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622: Bootstrap starting.
I20260925 13:21:32.509012 22109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:32.509265 22109 log.cc:824] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:32.510010 22109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622: No bootstrap required, opened a new log
I20260925 13:21:32.510844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 22041
I20260925 13:21:32.511077 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:32.511269 392 external_mini_cluster.cc:1428] Setting key e55f0237360e081813edd7a28602f7b0
I20260925 13:21:32.512156 22109 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e58d126bc04e409f4332a2361f4622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } }
I20260925 13:21:32.512333 22109 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:32.512368 22109 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e58d126bc04e409f4332a2361f4622, State: Initialized, Role: FOLLOWER
I20260925 13:21:32.512506 22109 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e58d126bc04e409f4332a2361f4622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } }
I20260925 13:21:32.512588 22109 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:32.512609 22109 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:32.512655 22109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:32.513217 22109 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e58d126bc04e409f4332a2361f4622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } }
I20260925 13:21:32.513311 22109 leader_election.cc:302] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6e58d126bc04e409f4332a2361f4622; no voters:
I20260925 13:21:32.513465 22109 leader_election.cc:288] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:32.513506 22114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:32.513654 22114 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [term 1 LEADER]: Becoming Leader. State: Replica: f6e58d126bc04e409f4332a2361f4622, State: Running, Role: LEADER
I20260925 13:21:32.513734 22109 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:32.513789 22114 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e58d126bc04e409f4332a2361f4622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } }
I20260925 13:21:32.514139 22114 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6e58d126bc04e409f4332a2361f4622. Latest consensus state: current_term: 1 leader_uuid: "f6e58d126bc04e409f4332a2361f4622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e58d126bc04e409f4332a2361f4622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } }
I20260925 13:21:32.514220 22114 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:32.514153 22115 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6e58d126bc04e409f4332a2361f4622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e58d126bc04e409f4332a2361f4622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33217 } } }
I20260925 13:21:32.514369 22115 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:32.515082 22121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:32.515533 22121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:32.517119 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:32.511812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45665 (local address 127.0.98.62:33217)
0925 13:21:32.512049 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:32.512060 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:32.512097 (+ 37us) server_negotiation.cc:407] Connection header received
0925 13:21:32.512217 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:32.512223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:32.512280 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:32.512428 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:32.512705 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:32.514141 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:32.515139 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:32.515558 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:32.515700 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:32.516356 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:32.516360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:32.516361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:32.516655 (+ 294us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:32.516696 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:32.516708 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:32.516778 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:32.516869 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 13:21:32.516899 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":105,"threads_started":1}
I20260925 13:21:32.517586 22121 catalog_manager.cc:1347] Generated new cluster ID: d626e0465e854077b963be5c16de1ad0
I20260925 13:21:32.517632 22121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:32.521903 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:32.532544 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:32.523235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60221 (local address 127.0.98.62:33217)
0925 13:21:32.523274 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:32.523277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:32.529792 (+ 6515us) server_negotiation.cc:407] Connection header received
0925 13:21:32.529896 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:32.529897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:32.529910 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:32.529953 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:32.530100 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:32.530594 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:32.531316 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:32.531545 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:32.531604 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:32.532291 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:32.532292 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:32.532293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:32.532298 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:32.532315 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:32.532320 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:32.532362 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:32.532438 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 13:21:32.532451 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:21:32.532953 22121 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:32.533695 22121 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:32.542793 22121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6e58d126bc04e409f4332a2361f4622: Generated new TSK 0
I20260925 13:21:32.543013 22121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:32.549106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 22041
W20260925 13:21:32.552932 22026 connection.cc:535] client connection to 127.0.98.62:33217 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:33217): Connection reset by peer (error 104)
2026-09-25T13:21:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:32 dist-test-slave-t8d9 krb5kdc[22140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:32 dist-test-slave-t8d9 krb5kdc[22140](info): set up 2 sockets
Sep 25 13:21:32 dist-test-slave-t8d9 krb5kdc[22140](info): commencing operation
krb5kdc: starting...
W20260925 13:21:33.493324 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:33 dist-test-slave-t8d9 krb5kdc[22140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:21:33.607860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40289
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:33.687106 22153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:33.687326 22153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:33.687348 22153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:33.688499 22153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:33.688542 22153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:33.688555 22153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:33.688566 22153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:33.688576 22153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:33.690075 22153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40289
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:33.690295 22153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:33.690508 22153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:33.695122 22159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:33.695147 22158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:33.695122 22161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:33.695493 22153 server_base.cc:1034] running on GCE node
I20260925 13:21:33.695765 22153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:33.696167 22153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:33.697326 22153 hybrid_clock.cc:648] HybridClock initialized: now 1790342493697308 us; error 34 us; skew 500 ppm
Sep 25 13:21:33 dist-test-slave-t8d9 krb5kdc[22140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:33.699721 22153 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:21:33.700177 22153 webserver.cc:460] Webserver started at http://127.0.98.62:46155/ using document root <none> and password file <none>
I20260925 13:21:33.700389 22153 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:33.700433 22153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:33.700552 22153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:33.701645 22153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "17524c605d214b8bbceb0e065dc6bac3"
format_stamp: "Formatted at 2026-09-25 13:21:33 on dist-test-slave-t8d9"
server_key: "ec5e5e1d1a7b0da385d7784660c33e1e"
server_key_iv: "0a306c9d3c127b768de9f772cb22bea3"
server_key_version: "encryptionkey@0"
I20260925 13:21:33.701972 22153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "17524c605d214b8bbceb0e065dc6bac3"
format_stamp: "Formatted at 2026-09-25 13:21:33 on dist-test-slave-t8d9"
server_key: "ec5e5e1d1a7b0da385d7784660c33e1e"
server_key_iv: "0a306c9d3c127b768de9f772cb22bea3"
server_key_version: "encryptionkey@0"
I20260925 13:21:33.703168 22153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:21:33.703789 22168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:33.703924 22153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:33.703995 22153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "17524c605d214b8bbceb0e065dc6bac3"
format_stamp: "Formatted at 2026-09-25 13:21:33 on dist-test-slave-t8d9"
server_key: "ec5e5e1d1a7b0da385d7784660c33e1e"
server_key_iv: "0a306c9d3c127b768de9f772cb22bea3"
server_key_version: "encryptionkey@0"
I20260925 13:21:33.704061 22153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:33.707244 22153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:33.708870 22153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:33.708993 22153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:33.712606 22153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40289
I20260925 13:21:33.712634 22220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40289 every 8 connection(s)
I20260925 13:21:33.712920 22153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:33.713827 22221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:33.715942 22221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3: Bootstrap starting.
I20260925 13:21:33.716455 22221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:33.716692 22221 log.cc:824] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:33.717283 22221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3: No bootstrap required, opened a new log
I20260925 13:21:33.718823 22221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17524c605d214b8bbceb0e065dc6bac3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40289 } }
I20260925 13:21:33.718953 22221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:33.718974 22221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17524c605d214b8bbceb0e065dc6bac3, State: Initialized, Role: FOLLOWER
I20260925 13:21:33.719094 22221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17524c605d214b8bbceb0e065dc6bac3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40289 } }
I20260925 13:21:33.719153 22221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:33.719173 22221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:33.719218 22221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:33.719694 22221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17524c605d214b8bbceb0e065dc6bac3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40289 } }
I20260925 13:21:33.719779 22221 leader_election.cc:302] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17524c605d214b8bbceb0e065dc6bac3; no voters:
I20260925 13:21:33.719920 22221 leader_election.cc:288] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:33.719959 22224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:33.720120 22224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [term 1 LEADER]: Becoming Leader. State: Replica: 17524c605d214b8bbceb0e065dc6bac3, State: Running, Role: LEADER
I20260925 13:21:33.720146 22221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:33.720206 22224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17524c605d214b8bbceb0e065dc6bac3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40289 } }
I20260925 13:21:33.720788 22226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17524c605d214b8bbceb0e065dc6bac3. Latest consensus state: current_term: 1 leader_uuid: "17524c605d214b8bbceb0e065dc6bac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17524c605d214b8bbceb0e065dc6bac3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40289 } } }
I20260925 13:21:33.720813 22225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17524c605d214b8bbceb0e065dc6bac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17524c605d214b8bbceb0e065dc6bac3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40289 } } }
I20260925 13:21:33.720865 22226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:33.720891 22225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:33.721048 22233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:33.721513 22233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:33.722289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 22153
I20260925 13:21:33.722393 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:33.722573 392 external_mini_cluster.cc:1428] Setting key c674743730512789affd526c4ae91434
I20260925 13:21:33.722867 22233 catalog_manager.cc:1347] Generated new cluster ID: b7d0b4a4389c4cd69ec4e79c1c882915
I20260925 13:21:33.722911 22233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:33.727478 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:33.722995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55631 (local address 127.0.98.62:40289)
0925 13:21:33.725900 (+ 2905us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:33.725908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:33.725935 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:21:33.726036 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:33.726042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:33.726070 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:33.726185 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:33.726844 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:33.726847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:33.726847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:33.726850 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:33.726887 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:33.726900 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:33.726954 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:33.727048 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 13:21:33.727162 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2866,"thread_start_us":83,"threads_started":1}
I20260925 13:21:33.728287 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:33.730690 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:33.729811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60603 (local address 127.0.98.62:40289)
0925 13:21:33.729848 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:33.729850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:33.729857 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:21:33.729877 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:33.729878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:33.729883 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:33.729937 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:33.730495 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:33.730496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:33.730497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:33.730501 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:33.730516 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:33.730521 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:33.730575 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:33.730616 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 13:21:33.730628 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:21:33.733767 22233 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:33.733907 22233 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:33.738197 22233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17524c605d214b8bbceb0e065dc6bac3: Generated new TSK 0
I20260925 13:21:33.738369 22233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:33.742364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 22153
2026-09-25T13:21:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1193 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:33 dist-test-slave-t8d9 krb5kdc[22252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:33 dist-test-slave-t8d9 krb5kdc[22252](info): set up 2 sockets
Sep 25 13:21:33 dist-test-slave-t8d9 krb5kdc[22252](info): commencing operation
krb5kdc: starting...
W20260925 13:21:34.697429 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:34 dist-test-slave-t8d9 krb5kdc[22252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:21:34.810035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44761
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:34.889605 22265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:34.889825 22265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:34.889854 22265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:34.891031 22265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:34.891075 22265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:34.891093 22265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:34.891109 22265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:34.891124 22265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:34.892625 22265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:34.892869 22265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:34.893079 22265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:34.897523 22273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:34.897589 22271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:34.897589 22270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:34.897753 22265 server_base.cc:1034] running on GCE node
I20260925 13:21:34.897960 22265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:34.898195 22265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:34.899338 22265 hybrid_clock.cc:648] HybridClock initialized: now 1790342494899319 us; error 28 us; skew 500 ppm
Sep 25 13:21:34 dist-test-slave-t8d9 krb5kdc[22252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:34.905964 22265 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:21:34.906416 22265 webserver.cc:460] Webserver started at http://127.0.98.62:43743/ using document root <none> and password file <none>
I20260925 13:21:34.906633 22265 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:34.906680 22265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:34.906803 22265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:34.907883 22265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "107514599b544c13a62fa02820c1ea4d"
format_stamp: "Formatted at 2026-09-25 13:21:34 on dist-test-slave-t8d9"
server_key: "e541f26e345e229ada8cfeeca09aa2fb"
server_key_iv: "16f5650037acf9b54861976c862afbf3"
server_key_version: "encryptionkey@0"
I20260925 13:21:34.908203 22265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "107514599b544c13a62fa02820c1ea4d"
format_stamp: "Formatted at 2026-09-25 13:21:34 on dist-test-slave-t8d9"
server_key: "e541f26e345e229ada8cfeeca09aa2fb"
server_key_iv: "16f5650037acf9b54861976c862afbf3"
server_key_version: "encryptionkey@0"
I20260925 13:21:34.909404 22265 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:21:34.910095 22280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:34.910238 22265 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:34.910322 22265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "107514599b544c13a62fa02820c1ea4d"
format_stamp: "Formatted at 2026-09-25 13:21:34 on dist-test-slave-t8d9"
server_key: "e541f26e345e229ada8cfeeca09aa2fb"
server_key_iv: "16f5650037acf9b54861976c862afbf3"
server_key_version: "encryptionkey@0"
I20260925 13:21:34.910388 22265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:34.921185 22265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:34.922955 22265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:34.923086 22265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:34.926780 22265 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44761
I20260925 13:21:34.926815 22332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44761 every 8 connection(s)
I20260925 13:21:34.927138 22265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:34.927937 22333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:34.930169 22333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d: Bootstrap starting.
I20260925 13:21:34.930707 22333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:34.930960 22333 log.cc:824] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:34.931691 22333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d: No bootstrap required, opened a new log
I20260925 13:21:34.933241 22333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107514599b544c13a62fa02820c1ea4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44761 } }
I20260925 13:21:34.933378 22333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:34.933399 22333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 107514599b544c13a62fa02820c1ea4d, State: Initialized, Role: FOLLOWER
I20260925 13:21:34.933558 22333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107514599b544c13a62fa02820c1ea4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44761 } }
I20260925 13:21:34.933624 22333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:34.933643 22333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:34.933677 22333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:34.934178 22333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107514599b544c13a62fa02820c1ea4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44761 } }
I20260925 13:21:34.934264 22333 leader_election.cc:302] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 107514599b544c13a62fa02820c1ea4d; no voters:
I20260925 13:21:34.934412 22333 leader_election.cc:288] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:34.934453 22336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:34.934598 22336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [term 1 LEADER]: Becoming Leader. State: Replica: 107514599b544c13a62fa02820c1ea4d, State: Running, Role: LEADER
I20260925 13:21:34.934685 22336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107514599b544c13a62fa02820c1ea4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44761 } }
I20260925 13:21:34.934700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 22265
I20260925 13:21:34.934791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:34.934967 392 external_mini_cluster.cc:1428] Setting key cf6bd8441e7408b0f0a6d4c68ab088d1
I20260925 13:21:34.935161 22337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "107514599b544c13a62fa02820c1ea4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107514599b544c13a62fa02820c1ea4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44761 } } }
I20260925 13:21:34.935241 22337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:34.935254 22333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:34.935704 22338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 107514599b544c13a62fa02820c1ea4d. Latest consensus state: current_term: 1 leader_uuid: "107514599b544c13a62fa02820c1ea4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107514599b544c13a62fa02820c1ea4d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44761 } } }
I20260925 13:21:34.935779 22338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:34.935918 22340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:34.936408 22340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:34.937810 22340 catalog_manager.cc:1347] Generated new cluster ID: 2d941b1eba7849b69147a5402161aecf
I20260925 13:21:34.937855 22340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:21:34 dist-test-slave-t8d9 krb5kdc[22252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:34.946305 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:34.935539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56791 (local address 127.0.98.62:44761)
0925 13:21:34.935693 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:34.935704 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:34.935863 (+ 159us) server_negotiation.cc:407] Connection header received
0925 13:21:34.936005 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:34.936015 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:34.936076 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:34.936222 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:34.936939 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:34.938304 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:34.939334 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:34.939946 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:34.940062 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:34.942732 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:34.942736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:34.942737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:34.942748 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:34.944630 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:34.945021 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:34.945025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:34.945025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:34.945135 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:34.945321 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:34.945321 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:34.945322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:34.945677 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:34.945824 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:34.945939 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 13:21:34.946077 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260925 13:21:34.946857 22340 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:34.947252 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:34.947494 22340 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:34.955875 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:34.948777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44749 (local address 127.0.98.62:44761)
0925 13:21:34.948807 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:34.948808 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:34.948815 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:21:34.948867 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:34.948867 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:34.948877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:34.948926 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:34.949516 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:34.950280 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:34.951345 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:34.951661 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:34.951721 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:34.953140 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:34.953141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:34.953142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:34.953149 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:34.954699 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:34.955083 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:34.955084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:34.955084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:34.955135 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:34.955306 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:34.955307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:34.955307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:34.955419 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:34.955503 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:34.955605 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 13:21:34.955683 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 13:21:34.960472 22340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 107514599b544c13a62fa02820c1ea4d: Generated new TSK 0
I20260925 13:21:34.960661 22340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:34.964156 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 22265
2026-09-25T13:21:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:35 dist-test-slave-t8d9 krb5kdc[22364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:35 dist-test-slave-t8d9 krb5kdc[22364](info): set up 2 sockets
Sep 25 13:21:35 dist-test-slave-t8d9 krb5kdc[22364](info): commencing operation
krb5kdc: starting...
W20260925 13:21:35.909417 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:35 dist-test-slave-t8d9 krb5kdc[22364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:21:36.028142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:36.109361 22377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:36.109629 22377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:36.109665 22377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:36.110873 22377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:36.110926 22377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:36.110945 22377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:36.110963 22377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:36.110985 22377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:36.112562 22377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:36.112855 22377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:36.113090 22377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:36.117837 22385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:36.117871 22383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:36.117870 22382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:36.118074 22377 server_base.cc:1034] running on GCE node
I20260925 13:21:36.118328 22377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:36.118603 22377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:36.119740 22377 hybrid_clock.cc:648] HybridClock initialized: now 1790342496119727 us; error 30 us; skew 500 ppm
Sep 25 13:21:36 dist-test-slave-t8d9 krb5kdc[22364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:36.122239 22377 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:21:36.122696 22377 webserver.cc:460] Webserver started at http://127.0.98.62:45981/ using document root <none> and password file <none>
I20260925 13:21:36.122888 22377 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:36.122927 22377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:36.123029 22377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:36.124060 22377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "f4449ae0fc33444db751076651d3ce18"
format_stamp: "Formatted at 2026-09-25 13:21:36 on dist-test-slave-t8d9"
server_key: "59bbff1abdef7ef9d8c5931d070cc68f"
server_key_iv: "6a64010e2cac99d9b59d9659b27d2b19"
server_key_version: "encryptionkey@0"
I20260925 13:21:36.124351 22377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4449ae0fc33444db751076651d3ce18"
format_stamp: "Formatted at 2026-09-25 13:21:36 on dist-test-slave-t8d9"
server_key: "59bbff1abdef7ef9d8c5931d070cc68f"
server_key_iv: "6a64010e2cac99d9b59d9659b27d2b19"
server_key_version: "encryptionkey@0"
I20260925 13:21:36.125593 22377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:21:36.126298 22392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:36.126438 22377 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:36.126523 22377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "f4449ae0fc33444db751076651d3ce18"
format_stamp: "Formatted at 2026-09-25 13:21:36 on dist-test-slave-t8d9"
server_key: "59bbff1abdef7ef9d8c5931d070cc68f"
server_key_iv: "6a64010e2cac99d9b59d9659b27d2b19"
server_key_version: "encryptionkey@0"
I20260925 13:21:36.126582 22377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:36.169296 22377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:36.171396 22377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:36.171515 22377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:36.175483 22377 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41741
I20260925 13:21:36.175519 22444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41741 every 8 connection(s)
I20260925 13:21:36.175846 22377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:36.176751 22445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:36.179014 22445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18: Bootstrap starting.
I20260925 13:21:36.179553 22445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:36.179775 22445 log.cc:824] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:36.180550 22445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18: No bootstrap required, opened a new log
I20260925 13:21:36.182185 22445 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4449ae0fc33444db751076651d3ce18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41741 } }
I20260925 13:21:36.182330 22445 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:36.182351 22445 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4449ae0fc33444db751076651d3ce18, State: Initialized, Role: FOLLOWER
I20260925 13:21:36.182473 22445 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4449ae0fc33444db751076651d3ce18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41741 } }
I20260925 13:21:36.182548 22445 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:36.182578 22445 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:36.182610 22445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:36.183102 22445 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4449ae0fc33444db751076651d3ce18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41741 } }
I20260925 13:21:36.183185 22445 leader_election.cc:302] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4449ae0fc33444db751076651d3ce18; no voters:
I20260925 13:21:36.183319 22445 leader_election.cc:288] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:36.183398 22448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:36.183552 22445 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:36.183559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 22377
I20260925 13:21:36.183604 22448 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [term 1 LEADER]: Becoming Leader. State: Replica: f4449ae0fc33444db751076651d3ce18, State: Running, Role: LEADER
I20260925 13:21:36.183657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:36.183692 22448 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4449ae0fc33444db751076651d3ce18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41741 } }
I20260925 13:21:36.183844 392 external_mini_cluster.cc:1428] Setting key 7391d53097c554d3f2efb9372d26eca5
I20260925 13:21:36.184254 22449 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4449ae0fc33444db751076651d3ce18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4449ae0fc33444db751076651d3ce18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41741 } } }
I20260925 13:21:36.184335 22449 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:36.184305 22450 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4449ae0fc33444db751076651d3ce18. Latest consensus state: current_term: 1 leader_uuid: "f4449ae0fc33444db751076651d3ce18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4449ae0fc33444db751076651d3ce18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41741 } } }
I20260925 13:21:36.184378 22450 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:36.184636 22459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:36.185176 22459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:36.186866 22459 catalog_manager.cc:1347] Generated new cluster ID: 4423df5baed444eb923ee2371e0115f2
I20260925 13:21:36.186918 22459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:36.190112 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:36.184304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48855 (local address 127.0.98.62:41741)
0925 13:21:36.185025 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:36.185035 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:36.185065 (+ 30us) server_negotiation.cc:407] Connection header received
0925 13:21:36.185187 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:36.185193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:36.185246 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:36.185681 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:36.185758 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:36.186884 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:36.188028 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:36.188632 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:36.188733 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:36.189417 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:36.189421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:36.189421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:36.189649 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:36.189686 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:36.189701 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:36.189754 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:36.189840 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 13:21:36.189897 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":63,"threads_started":1}
I20260925 13:21:36.191162 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:36.194701 22459 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:36.195420 22459 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:36.196103 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:36.192923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54143 (local address 127.0.98.62:41741)
0925 13:21:36.192966 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:36.192968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:36.193177 (+ 209us) server_negotiation.cc:407] Connection header received
0925 13:21:36.193227 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:36.193228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:36.193239 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:36.193289 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:36.193429 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:36.194126 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:36.194973 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:36.195199 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:36.195256 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:36.195854 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:36.195855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:36.195856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:36.195861 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:36.195876 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:36.195880 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:36.195924 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:36.195983 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 13:21:36.196027 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":17}
I20260925 13:21:36.203686 22459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4449ae0fc33444db751076651d3ce18: Generated new TSK 0
I20260925 13:21:36.203920 22459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:36.208669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 22377
2026-09-25T13:21:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1244 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:36 dist-test-slave-t8d9 krb5kdc[22476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:36 dist-test-slave-t8d9 krb5kdc[22476](info): set up 2 sockets
Sep 25 13:21:36 dist-test-slave-t8d9 krb5kdc[22476](info): commencing operation
krb5kdc: starting...
W20260925 13:21:37.169571 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:37 dist-test-slave-t8d9 krb5kdc[22476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:21:37.282310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37065
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:37.360826 22489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:37.361052 22489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:37.361075 22489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:37.362241 22489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:37.362282 22489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:37.362295 22489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:37.362306 22489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:37.362326 22489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:37.363767 22489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:37.363991 22489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:37.364187 22489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 13:21:37.368676 22489 server_base.cc:1034] running on GCE node
W20260925 13:21:37.368654 22495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:37.368647 22497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:37.368634 22494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:37.369072 22489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:37.369330 22489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:37.370460 22489 hybrid_clock.cc:648] HybridClock initialized: now 1790342497370449 us; error 27 us; skew 500 ppm
Sep 25 13:21:37 dist-test-slave-t8d9 krb5kdc[22476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:37.372704 22489 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:21:37.373144 22489 webserver.cc:460] Webserver started at http://127.0.98.62:33257/ using document root <none> and password file <none>
I20260925 13:21:37.373317 22489 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:37.373354 22489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:37.373449 22489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:37.374476 22489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "323d525b7e23499c9f71ed27a8f2c5a1"
format_stamp: "Formatted at 2026-09-25 13:21:37 on dist-test-slave-t8d9"
server_key: "f4b34c9419550b1d9e7d3059c889c3c2"
server_key_iv: "f111f93b1819b488937fc6da8a226f7c"
server_key_version: "encryptionkey@0"
I20260925 13:21:37.374755 22489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "323d525b7e23499c9f71ed27a8f2c5a1"
format_stamp: "Formatted at 2026-09-25 13:21:37 on dist-test-slave-t8d9"
server_key: "f4b34c9419550b1d9e7d3059c889c3c2"
server_key_iv: "f111f93b1819b488937fc6da8a226f7c"
server_key_version: "encryptionkey@0"
I20260925 13:21:37.375908 22489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:21:37.376543 22504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:37.376694 22489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:37.376775 22489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "323d525b7e23499c9f71ed27a8f2c5a1"
format_stamp: "Formatted at 2026-09-25 13:21:37 on dist-test-slave-t8d9"
server_key: "f4b34c9419550b1d9e7d3059c889c3c2"
server_key_iv: "f111f93b1819b488937fc6da8a226f7c"
server_key_version: "encryptionkey@0"
I20260925 13:21:37.376832 22489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:37.380041 22489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:37.382115 22489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:37.382220 22489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:37.385720 22489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37065
I20260925 13:21:37.385754 22556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37065 every 8 connection(s)
I20260925 13:21:37.386055 22489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:37.386631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 22489
I20260925 13:21:37.386756 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:37.386972 392 external_mini_cluster.cc:1428] Setting key de9966be337f2137b4571a73e2a3e9e8
I20260925 13:21:37.387235 22557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:37.389412 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:37.387482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46517 (local address 127.0.98.62:37065)
0925 13:21:37.387769 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:37.387778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:37.387806 (+ 28us) server_negotiation.cc:407] Connection header received
0925 13:21:37.388311 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:37.388319 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:37.388352 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:37.388502 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:37.389021 (+ 519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:37.389025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:37.389025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:37.389029 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:37.389072 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:37.389086 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:37.389156 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:37.389193 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 13:21:37.389231 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":71,"threads_started":1}
I20260925 13:21:37.390475 22557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1: Bootstrap starting.
I20260925 13:21:37.390487 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:37.391165 22557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:37.391484 22557 log.cc:824] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:37.392195 22557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1: No bootstrap required, opened a new log
I20260925 13:21:37.394049 22557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323d525b7e23499c9f71ed27a8f2c5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37065 } }
I20260925 13:21:37.394202 22557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 13:21:37.394234 22557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 323d525b7e23499c9f71ed27a8f2c5a1, State: Initialized, Role: FOLLOWER
I20260925 13:21:37.394356 22557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323d525b7e23499c9f71ed27a8f2c5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37065 } }
I20260925 13:21:37.394452 22557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 13:21:37.394495 22557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 13:21:37.394543 22557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:37.394661 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:37.391984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37291 (local address 127.0.98.62:37065)
0925 13:21:37.392016 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:37.392018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:37.393762 (+ 1744us) server_negotiation.cc:407] Connection header received
0925 13:21:37.393825 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:37.393826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:37.393831 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:37.393876 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:37.394434 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:37.394436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:37.394436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 13:21:37.394440 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:37.394453 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 13:21:37.394457 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:37.394503 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:37.394584 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 13:21:37.394598 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 13:21:37.395231 22557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323d525b7e23499c9f71ed27a8f2c5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37065 } }
I20260925 13:21:37.395324 22557 leader_election.cc:302] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 323d525b7e23499c9f71ed27a8f2c5a1; no voters:
I20260925 13:21:37.395486 22557 leader_election.cc:288] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 13:21:37.395543 22567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:37.395676 22567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [term 1 LEADER]: Becoming Leader. State: Replica: 323d525b7e23499c9f71ed27a8f2c5a1, State: Running, Role: LEADER
I20260925 13:21:37.395714 22557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:37.395763 22567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323d525b7e23499c9f71ed27a8f2c5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37065 } }
I20260925 13:21:37.396119 22569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 323d525b7e23499c9f71ed27a8f2c5a1. Latest consensus state: current_term: 1 leader_uuid: "323d525b7e23499c9f71ed27a8f2c5a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323d525b7e23499c9f71ed27a8f2c5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37065 } } }
I20260925 13:21:37.396234 22569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:37.396116 22568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "323d525b7e23499c9f71ed27a8f2c5a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323d525b7e23499c9f71ed27a8f2c5a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37065 } } }
I20260925 13:21:37.396347 22568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:37.396811 22576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:37.397257 22576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:37.398690 22576 catalog_manager.cc:1347] Generated new cluster ID: 60880d7853f947f2be5a6c99efa25564
I20260925 13:21:37.398742 22576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:37.408788 22576 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:37.408981 22576 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:37.411388 22576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 323d525b7e23499c9f71ed27a8f2c5a1: Generated new TSK 0
I20260925 13:21:37.411612 22576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:37.415923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 22489
2026-09-25T13:21:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1207 ms)
[----------] 18 tests from AuthTokenIssuingTest (21849 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:37 dist-test-slave-t8d9 krb5kdc[22588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:37 dist-test-slave-t8d9 krb5kdc[22588](info): set up 2 sockets
Sep 25 13:21:37 dist-test-slave-t8d9 krb5kdc[22588](info): commencing operation
krb5kdc: starting...
W20260925 13:21:38.357391 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:21:38.466347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40499
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40499,127.0.98.61:41157,127.0.98.60:41343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:38.547009 22601 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 13:21:38.547247 22601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:38.547271 22601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:38.547297 22601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:38.548463 22601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:38.548506 22601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:38.548523 22601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:38.548538 22601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:38.548553 22601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:38.550093 22601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40499,127.0.98.61:41157,127.0.98.60:41343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:38.550359 22601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:38.550570 22601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:38.555090 22609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:38.555121 22606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:38.555188 22601 server_base.cc:1034] running on GCE node
W20260925 13:21:38.555107 22607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:38.555505 22601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:38.555766 22601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:38.556902 22601 hybrid_clock.cc:648] HybridClock initialized: now 1790342498556886 us; error 28 us; skew 500 ppm
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:38.559453 22601 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:21:38.559947 22601 webserver.cc:460] Webserver started at http://127.0.98.62:40803/ using document root <none> and password file <none>
I20260925 13:21:38.560134 22601 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:38.560173 22601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:38.560281 22601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:38.561301 22601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "0c02ac5b9034426aa4d547de0814f41a"
format_stamp: "Formatted at 2026-09-25 13:21:38 on dist-test-slave-t8d9"
server_key: "e4705532906e39155979a4aec6fceb89"
server_key_iv: "8b23e1e5a839a5ad782c0bfe8bdf9233"
server_key_version: "encryptionkey@0"
I20260925 13:21:38.561653 22601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c02ac5b9034426aa4d547de0814f41a"
format_stamp: "Formatted at 2026-09-25 13:21:38 on dist-test-slave-t8d9"
server_key: "e4705532906e39155979a4aec6fceb89"
server_key_iv: "8b23e1e5a839a5ad782c0bfe8bdf9233"
server_key_version: "encryptionkey@0"
I20260925 13:21:38.562875 22601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:21:38.563593 22616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:38.563820 22601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 13:21:38.563917 22601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "0c02ac5b9034426aa4d547de0814f41a"
format_stamp: "Formatted at 2026-09-25 13:21:38 on dist-test-slave-t8d9"
server_key: "e4705532906e39155979a4aec6fceb89"
server_key_iv: "8b23e1e5a839a5ad782c0bfe8bdf9233"
server_key_version: "encryptionkey@0"
I20260925 13:21:38.564147 22601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:38.573220 22601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:38.575218 22601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:38.575333 22601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:38.579137 22601 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40499
I20260925 13:21:38.579166 22668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40499 every 8 connection(s)
I20260925 13:21:38.579494 22601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:38.580446 22669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:38.580616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 22601
I20260925 13:21:38.580721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:38.580891 392 external_mini_cluster.cc:1428] Setting key ce5a7f18ba44133f73538e84ecd6c1a3
I20260925 13:21:38.582141 22669 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:38.593681 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.582855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40499 (local address 127.0.0.1:47696)
0925 13:21:38.583057 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.583095 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:38.583574 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:38.583840 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:38.583843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:38.584113 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:38.584265 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.584269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.585253 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.585254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.585984 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.585986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.586441 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.586442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.586505 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.586988 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:38.586997 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:38.590192 (+ 3195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:38.592094 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.592101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.592103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.592295 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.592542 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.592543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.592543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.592616 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.593069 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:38.593075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:38.593149 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 13:21:38.593239 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 13:21:38.593381 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
I20260925 13:21:38.593690 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.583044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.0.98.62:40499)
0925 13:21:38.583182 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:38.583187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:38.583219 (+ 32us) server_negotiation.cc:407] Connection header received
0925 13:21:38.583724 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:38.583726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:38.583746 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:38.583824 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:38.584353 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.585164 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.586056 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.586358 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.586459 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.590311 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:38.590319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:38.590319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:38.590326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:38.592017 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.592367 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.592370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.592370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.592481 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.592671 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.592672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.592673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.592979 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:38.593070 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:38.593283 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 13:21:38.593452 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260925 13:21:38.594467 22669 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:21:38.594806 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.594699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41157 (local address 127.0.0.1:55140)
0925 13:21:38.594740 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.594783 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 13:21:38.594945 22619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41157: connect: Connection refused (error 111)
I20260925 13:21:38.595346 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.595266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41157 (local address 127.0.0.1:55154)
0925 13:21:38.595306 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.595330 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 13:21:38.595448 22669 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41157: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41157: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 13:21:38.604040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41157
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40499,127.0.98.61:41157,127.0.98.60:41343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:21:38.644762 22669 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } attempt: 1
I20260925 13:21:38.645332 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.645175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41157 (local address 127.0.0.1:55156)
0925 13:21:38.645245 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.645298 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 13:21:38.645716 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.645610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41157 (local address 127.0.0.1:55172)
0925 13:21:38.645658 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.645691 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 13:21:38.645823 22669 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41157: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41157: connect: Connection refused (error 111)
W20260925 13:21:38.684370 22676 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 13:21:38.684577 22676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:38.684593 22676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:38.684612 22676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:38.685775 22676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:38.685815 22676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:38.685828 22676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:38.685839 22676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:38.685850 22676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:38.687319 22676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40499,127.0.98.61:41157,127.0.98.60:41343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:38.687537 22676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:38.687754 22676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:38.692204 22684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:38.692220 22681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:38.692426 22676 server_base.cc:1034] running on GCE node
W20260925 13:21:38.692238 22682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:38.692725 22676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:38.693035 22676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:38.694195 22676 hybrid_clock.cc:648] HybridClock initialized: now 1790342498694176 us; error 31 us; skew 500 ppm
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:38.696626 22676 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 13:21:38.697093 22676 webserver.cc:460] Webserver started at http://127.0.98.61:43227/ using document root <none> and password file <none>
I20260925 13:21:38.697307 22676 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:38.697367 22676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:38.697490 22676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:38.698596 22676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/data/instance:
uuid: "a5512f0b2e5f47bd96177089db95ab7b"
format_stamp: "Formatted at 2026-09-25 13:21:38 on dist-test-slave-t8d9"
server_key: "159deb0074d36a22fb07e824787cfd19"
server_key_iv: "762e164a7fa20c80aa81f2e7546cbbe7"
server_key_version: "encryptionkey@0"
I20260925 13:21:38.698925 22676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/wal/instance:
uuid: "a5512f0b2e5f47bd96177089db95ab7b"
format_stamp: "Formatted at 2026-09-25 13:21:38 on dist-test-slave-t8d9"
server_key: "159deb0074d36a22fb07e824787cfd19"
server_key_iv: "762e164a7fa20c80aa81f2e7546cbbe7"
server_key_version: "encryptionkey@0"
I20260925 13:21:38.700181 22676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:21:38.700919 22691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:38.701081 22676 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:38.701171 22676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/wal
uuid: "a5512f0b2e5f47bd96177089db95ab7b"
format_stamp: "Formatted at 2026-09-25 13:21:38 on dist-test-slave-t8d9"
server_key: "159deb0074d36a22fb07e824787cfd19"
server_key_iv: "762e164a7fa20c80aa81f2e7546cbbe7"
server_key_version: "encryptionkey@0"
I20260925 13:21:38.701231 22676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:38.713276 22676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:38.713929 22669 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } attempt: 2
I20260925 13:21:38.714309 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.714200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41157 (local address 127.0.0.1:55188)
0925 13:21:38.714251 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.714277 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 13:21:38.714675 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.714576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41157 (local address 127.0.0.1:55202)
0925 13:21:38.714624 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.714651 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 13:21:38.715018 22669 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41157: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41157: connect: Connection refused (error 111)
I20260925 13:21:38.715765 22676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:38.715876 22676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:38.719486 22676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41157
I20260925 13:21:38.719520 22743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41157 every 8 connection(s)
I20260925 13:21:38.719823 22676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/data/info.pb
I20260925 13:21:38.720721 22744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:38.722213 22744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:38.728699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 22676
I20260925 13:21:38.728798 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-1/wal/instance
I20260925 13:21:38.728986 392 external_mini_cluster.cc:1428] Setting key 3fb7c12a5ef94008d12dc20e5256d733
I20260925 13:21:38.730706 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.722804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47708 (local address 127.0.98.62:40499)
0925 13:21:38.722851 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:38.722853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:38.723016 (+ 163us) server_negotiation.cc:407] Connection header received
0925 13:21:38.723470 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:38.723471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:38.723482 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:38.723529 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:38.724101 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.724582 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.725500 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.725761 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.725817 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.727760 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:38.727762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:38.727762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:38.727769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:38.729240 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.729710 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.729711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.729712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.729766 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.730034 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.730035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.730035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.730159 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:38.730238 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:38.730526 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 13:21:38.730616 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 13:21:38.730892 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.722775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40499 (local address 127.0.0.1:47708)
0925 13:21:38.722911 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.722944 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:38.723373 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:38.723601 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:38.723604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:38.723830 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:38.724019 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.724023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.724658 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.724659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.725424 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.725425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.725839 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.725840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.725952 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.726329 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:38.726337 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:38.727677 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:38.729314 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.729321 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.729324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.729643 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.729823 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.729824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.729824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.729949 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.730248 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:38.730256 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:38.730400 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 13:21:38.730495 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 13:21:38.730671 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
I20260925 13:21:38.731482 22744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260925 13:21:38.739187 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.731731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41157 (local address 127.0.0.1:55216)
0925 13:21:38.731781 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.731792 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:38.731837 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:38.732089 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:38.732090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:38.732237 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:38.732298 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.732300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.733195 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.733196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.734005 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.734008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.734544 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.734545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.734623 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.735010 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:38.735015 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:38.736407 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:38.738095 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.738096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.738096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.738307 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.738510 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.738512 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.738512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.738589 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.738879 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:38.738880 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:38.738929 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 13:21:38.739008 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 13:21:38.739112 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 13:21:38.739372 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.731790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.0.98.61:41157)
0925 13:21:38.731968 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:38.731972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:38.731986 (+ 14us) server_negotiation.cc:407] Connection header received
0925 13:21:38.732004 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:38.732005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:38.732016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:38.732072 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:38.732365 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.733116 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.734101 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.734465 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.734533 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.736512 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:38.736520 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:38.736520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:38.736527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:38.738007 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.738371 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.738373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.738374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.738428 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.738655 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.738655 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.738656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.738817 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:38.738878 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:38.739048 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 13:21:38.739174 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
I20260925 13:21:38.740078 22744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:21:38.740445 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.740337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41343 (local address 127.0.0.1:45032)
0925 13:21:38.740382 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.740427 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 13:21:38.740573 22692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
I20260925 13:21:38.740975 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.740869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41343 (local address 127.0.0.1:45042)
0925 13:21:38.740937 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.740959 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 13:21:38.741074 22744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 13:21:38.752604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41343
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40499,127.0.98.61:41157,127.0.98.60:41343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:21:38.790319 22744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } attempt: 1
I20260925 13:21:38.790807 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.790679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41343 (local address 127.0.0.1:45056)
0925 13:21:38.790735 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.790776 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 13:21:38.791177 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.791040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41343 (local address 127.0.0.1:45062)
0925 13:21:38.791100 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.791145 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 13:21:38.791285 22744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
I20260925 13:21:38.806177 22669 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } attempt: 3
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:21:38.813304 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.806570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41157 (local address 127.0.0.1:55220)
0925 13:21:38.806648 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.806660 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:38.806724 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:38.806864 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:38.806865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:38.807021 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:38.807085 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.807087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.808022 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.808024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.808561 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.808563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.809045 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.809046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.809116 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.809566 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:38.809571 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:38.810733 (+ 1162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:38.812321 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.812322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.812323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.812489 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.812665 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.812665 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.812666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.812719 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.813034 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:38.813035 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:38.813082 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 13:21:38.813149 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 13:21:38.813230 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 13:21:38.813364 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.806656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.0.98.61:41157)
0925 13:21:38.806724 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:38.806727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:38.806761 (+ 34us) server_negotiation.cc:407] Connection header received
0925 13:21:38.806787 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:38.806788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:38.806802 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:38.806847 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:38.807153 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.807924 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.808637 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.808953 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.809022 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.810806 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:38.810807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:38.810807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:38.810815 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:38.812234 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.812554 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.812555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.812555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.812612 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.812768 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.812769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.812769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.812894 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:38.812958 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:38.813186 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 13:21:38.813266 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 13:21:38.813810 22669 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:21:38.814116 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.814028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41343 (local address 127.0.0.1:45074)
0925 13:21:38.814068 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.814096 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 13:21:38.814422 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.814337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41343 (local address 127.0.0.1:45078)
0925 13:21:38.814383 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.814408 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 13:21:38.814497 22669 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
W20260925 13:21:38.833776 22751 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 13:21:38.834008 22751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:38.834031 22751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:38.834060 22751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:38.835197 22751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:38.835237 22751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:38.835255 22751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:38.835270 22751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:38.835286 22751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:38.836738 22751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40499,127.0.98.61:41157,127.0.98.60:41343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:38.836980 22751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:38.837177 22751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:38.841665 22759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:38.841679 22757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:38.841854 22751 server_base.cc:1034] running on GCE node
W20260925 13:21:38.841687 22756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:38.842120 22751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:38.842371 22751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:38.843511 22751 hybrid_clock.cc:648] HybridClock initialized: now 1790342498843497 us; error 32 us; skew 500 ppm
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:38.845603 22669 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } attempt: 1
I20260925 13:21:38.845958 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.845849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41343 (local address 127.0.0.1:45088)
0925 13:21:38.845899 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.845930 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 13:21:38.846016 22751 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 13:21:38.846287 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.846196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41343 (local address 127.0.0.1:45104)
0925 13:21:38.846235 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.846261 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 13:21:38.846382 22669 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
I20260925 13:21:38.846575 22751 webserver.cc:460] Webserver started at http://127.0.98.60:37737/ using document root <none> and password file <none>
I20260925 13:21:38.846824 22751 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:38.846870 22751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:38.847010 22751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:38.848073 22751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/data/instance:
uuid: "01e4aa0c048f405fac3901eb7f9eb70d"
format_stamp: "Formatted at 2026-09-25 13:21:38 on dist-test-slave-t8d9"
server_key: "d1252c67d34f46f5f6034c13fa007cf6"
server_key_iv: "90e060e5f2f3b2eb7d30429ea314b807"
server_key_version: "encryptionkey@0"
I20260925 13:21:38.848412 22751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/wal/instance:
uuid: "01e4aa0c048f405fac3901eb7f9eb70d"
format_stamp: "Formatted at 2026-09-25 13:21:38 on dist-test-slave-t8d9"
server_key: "d1252c67d34f46f5f6034c13fa007cf6"
server_key_iv: "90e060e5f2f3b2eb7d30429ea314b807"
server_key_version: "encryptionkey@0"
I20260925 13:21:38.849751 22751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 13:21:38.850494 22766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:38.850630 22751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:38.850713 22751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/wal
uuid: "01e4aa0c048f405fac3901eb7f9eb70d"
format_stamp: "Formatted at 2026-09-25 13:21:38 on dist-test-slave-t8d9"
server_key: "d1252c67d34f46f5f6034c13fa007cf6"
server_key_iv: "90e060e5f2f3b2eb7d30429ea314b807"
server_key_version: "encryptionkey@0"
I20260925 13:21:38.850781 22751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:38.859427 22744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } attempt: 2
I20260925 13:21:38.859865 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.859719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41343 (local address 127.0.0.1:45108)
0925 13:21:38.859779 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.859826 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 13:21:38.860196 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.860104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41343 (local address 127.0.0.1:45110)
0925 13:21:38.860148 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.860172 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 13:21:38.860283 22744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41343: connect: Connection refused (error 111)
I20260925 13:21:38.861447 22751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:38.863623 22751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:38.863749 22751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:38.867743 22751 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41343
I20260925 13:21:38.867802 22818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41343 every 8 connection(s)
I20260925 13:21:38.868177 22751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/data/info.pb
I20260925 13:21:38.869119 22819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:38.870669 22819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:38.877455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 22751
I20260925 13:21:38.877574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790342239834770-392-0/minicluster-data/master-2/wal/instance
I20260925 13:21:38.877746 392 external_mini_cluster.cc:1428] Setting key fb0f064df9656cdfdc296639d02a56dc
I20260925 13:21:38.879619 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.871364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47712 (local address 127.0.98.62:40499)
0925 13:21:38.871417 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:38.871418 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:38.871579 (+ 161us) server_negotiation.cc:407] Connection header received
0925 13:21:38.872072 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:38.872073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:38.872083 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:38.872148 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:38.872730 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.873259 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.874360 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.874557 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.874621 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.876797 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:38.876799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:38.876799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:38.876806 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:38.878241 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.878673 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.878674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.878675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.878739 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.878958 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.878959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.878959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.879131 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:38.879185 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:38.879480 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 13:21:38.879548 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 13:21:38.879859 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.871291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40499 (local address 127.0.0.1:47712)
0925 13:21:38.871456 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.871498 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:38.871949 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:38.872210 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:38.872213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:38.872462 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:38.872663 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.872667 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.873353 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.873354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.874262 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.874265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.874664 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.874665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.874787 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.875317 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:38.875326 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:38.876695 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:38.878324 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.878328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.878331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.878607 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.878799 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.878799 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.878800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.878906 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.879188 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:38.879192 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:38.879352 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 13:21:38.879467 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 13:21:38.879625 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260925 13:21:38.880378 22819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:21:38.886606 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.878142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.0.98.60:41343)
0925 13:21:38.878282 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:38.878287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:38.878325 (+ 38us) server_negotiation.cc:407] Connection header received
0925 13:21:38.878383 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:38.878386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:38.878401 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:38.878481 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:38.879100 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.880124 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.881013 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.881452 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.881522 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.883899 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:38.883908 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:38.883909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:38.883916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:38.885372 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.885768 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.885771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.885771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.885818 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.886023 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.886024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.886024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.886224 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:38.886343 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:38.886435 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 13:21:38.886512 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260925 13:21:38.886998 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.880616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41157 (local address 127.0.0.1:55226)
0925 13:21:38.880656 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.880668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:38.880704 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:38.880842 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:38.880843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:38.880952 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:38.880997 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.880998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.881795 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.881796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.882520 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.882523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.882857 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.882858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.882933 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.883280 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:38.883284 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:38.884360 (+ 1076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:38.885986 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.885987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.885988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.886183 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.886372 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.886373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.886373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.886442 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.886728 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:38.886729 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:38.886772 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 13:21:38.886859 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 13:21:38.886941 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 13:21:38.887101 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.880694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.0.98.61:41157)
0925 13:21:38.880737 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:38.880739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:38.880746 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:21:38.880769 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:38.880770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:38.880779 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:38.880840 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:38.881049 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.881729 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.882604 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.882794 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.882841 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.884434 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:38.884436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:38.884436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:38.884443 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:38.885845 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.886254 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.886255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.886256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.886310 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.886513 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.886514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.886514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.886637 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:38.886698 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:38.886901 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 13:21:38.887009 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:21:38.887528 22819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:38.893648 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.887811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.0.98.60:41343)
0925 13:21:38.887847 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:38.887849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:38.887855 (+ 6us) server_negotiation.cc:407] Connection header received
0925 13:21:38.887872 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:38.887873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:38.887881 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:38.887940 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:38.888154 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.888636 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.889245 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.889573 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.889629 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.891387 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:38.891389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:38.891389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:38.891396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:38.892658 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.892974 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.892975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.892975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.893022 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.893174 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.893175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.893175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.893276 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:38.893327 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:38.893447 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 13:21:38.893524 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:21:38.893700 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.887753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41343 (local address 127.0.0.1:45122)
0925 13:21:38.887790 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.887798 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:38.887829 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:38.887942 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:38.887943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:38.888065 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:38.888093 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.888094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.888699 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.888700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.889182 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.889183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.889624 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.889625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.889672 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.890000 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:38.890004 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:38.891308 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:38.892730 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.892730 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.892731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.892874 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.893077 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.893078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.893078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.893126 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.893326 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:38.893327 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:38.893364 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 13:21:38.893421 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 13:21:38.893597 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":63}
I20260925 13:21:38.895005 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.887417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47720 (local address 127.0.98.62:40499)
0925 13:21:38.887465 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:38.887468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:38.887570 (+ 102us) server_negotiation.cc:407] Connection header received
0925 13:21:38.887623 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:38.887624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:38.887634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:38.887677 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:38.888230 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.888676 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.889472 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.889746 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.889811 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.892613 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:38.892615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:38.892616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:38.892624 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:38.894063 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.894393 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.894394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.894394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.894443 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.894621 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.894622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.894622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.894713 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:38.894815 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:38.894890 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 13:21:38.894944 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:21:38.895570 22819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d: Bootstrap starting.
I20260925 13:21:38.896569 22819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:38.896862 22819 log.cc:824] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:38.897692 22819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d: No bootstrap required, opened a new log
I20260925 13:21:38.899708 22819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } }
I20260925 13:21:38.899835 22819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e4aa0c048f405fac3901eb7f9eb70d, State: Initialized, Role: FOLLOWER
I20260925 13:21:38.899899 22819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } }
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:21:38.900167 22825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } } }
I20260925 13:21:38.900221 22819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:38.900250 22825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d [sys.catalog]: This master's current role is: FOLLOWER
W20260925 13:21:38.902099 22836 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:21:38.902153 22836 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:21:38.903208 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.895718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.0.98.61:41157)
0925 13:21:38.895756 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:38.895757 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:38.895765 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:21:38.895788 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:38.895789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:38.895798 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:38.895839 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:38.896387 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.896897 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.897756 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.897999 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.898053 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.900474 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:38.900475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:38.900476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:38.900483 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:38.902064 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.902467 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.902468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.902469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.902530 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.902707 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.902707 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.902708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.902836 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:38.903007 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:38.903046 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 13:21:38.903121 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 13:21:38.904729 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:38.902532 22751 master_runner.cc:186] Error getting master registration for 127.0.98.62:40499: OK, instance_id { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" instance_seqno: 1790342498578034 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 13:21:38.905350 22751 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 13:21:38.905939 22643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0c02ac5b9034426aa4d547de0814f41a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47720
I20260925 13:21:38.921521 22669 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } attempt: 2
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:21:38.929234 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.921907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41343 (local address 127.0.0.1:45132)
0925 13:21:38.922020 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.922036 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:38.922111 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:38.922284 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:38.922285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:38.922482 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:38.922538 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.922541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.923435 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.923436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.924257 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.924260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.924716 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.924717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.924779 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.925248 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:38.925253 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:38.926787 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:38.928265 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.928266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.928266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.928433 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.928606 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.928607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.928607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.928664 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.928920 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:38.928921 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:38.928970 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 13:21:38.929030 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 13:21:38.929134 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 13:21:38.929234 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.921996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.0.98.60:41343)
0925 13:21:38.922052 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:38.922054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:38.922104 (+ 50us) server_negotiation.cc:407] Connection header received
0925 13:21:38.922172 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:38.922173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:38.922183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:38.922268 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:38.922615 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.923357 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.924340 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.924646 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.924705 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.926889 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:38.926890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:38.926890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:38.926899 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:38.928180 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.928503 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.928504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.928505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.928555 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.928713 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.928714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.928714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.928862 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:38.928920 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:38.929059 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 13:21:38.929136 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 13:21:38.930596 22669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a: Bootstrap starting.
I20260925 13:21:38.931218 22669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:38.931483 22669 log.cc:824] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:38.932152 22669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a: No bootstrap required, opened a new log
I20260925 13:21:38.933142 22669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } }
I20260925 13:21:38.933259 22669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c02ac5b9034426aa4d547de0814f41a, State: Initialized, Role: FOLLOWER
I20260925 13:21:38.933332 22669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } }
I20260925 13:21:38.933601 22839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } } }
I20260925 13:21:38.933655 22669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:38.933692 22839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:21:38.933832 22643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 13:21:38.933950 22643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:38.934748 22643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } }
W20260925 13:21:38.935434 22850 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:21:38.935482 22850 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:21:38.935670 22793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c02ac5b9034426aa4d547de0814f41a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "01e4aa0c048f405fac3901eb7f9eb70d"
I20260925 13:21:38.935799 22793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:38.936661 22793 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c02ac5b9034426aa4d547de0814f41a in term 1.
I20260925 13:21:38.936846 22643 leader_election.cc:288] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [CANDIDATE]: Term 1 election: Requested vote from peers a5512f0b2e5f47bd96177089db95ab7b (127.0.98.61:41157), 01e4aa0c048f405fac3901eb7f9eb70d (127.0.98.60:41343)
I20260925 13:21:38.936899 22617 leader_election.cc:302] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e4aa0c048f405fac3901eb7f9eb70d, 0c02ac5b9034426aa4d547de0814f41a; no voters:
I20260925 13:21:38.937057 22839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:38.937194 22839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [term 1 LEADER]: Becoming Leader. State: Replica: 0c02ac5b9034426aa4d547de0814f41a, State: Running, Role: LEADER
I20260925 13:21:38.937273 22839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } }
I20260925 13:21:38.937711 22839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c02ac5b9034426aa4d547de0814f41a. Latest consensus state: current_term: 1 leader_uuid: "0c02ac5b9034426aa4d547de0814f41a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } } }
I20260925 13:21:38.937774 22839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:38.936295 22601 master_runner.cc:186] Error getting master registration for 127.0.98.61:41157: OK, instance_id { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" instance_seqno: 1790342498718489 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 13:21:38.939116 22601 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 13:21:38.945712 22854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:38.945960 22854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:38.951306 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.939398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.0.98.60:41343)
0925 13:21:38.939439 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:38.939441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:38.940340 (+ 899us) server_negotiation.cc:407] Connection header received
0925 13:21:38.941199 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:38.941201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:38.941212 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:38.941242 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:38.943535 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.944284 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.945188 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.945465 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.945507 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.948639 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:38.948641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:38.948641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:38.948649 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:38.950121 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.950487 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.950488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.950488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.950543 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.950744 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.950745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.950745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.950875 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:38.950943 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:38.951140 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 13:21:38.951215 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:21:38.951586 22744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } attempt: 3
I20260925 13:21:38.954092 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.939605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.0.98.62:40499)
0925 13:21:38.939645 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:38.939647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:38.940211 (+ 564us) server_negotiation.cc:407] Connection header received
0925 13:21:38.940385 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:38.940386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:38.940395 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:38.940422 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:38.941922 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.942585 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.945871 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.946169 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.946212 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.948350 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:38.948351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:38.948351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:38.948359 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:38.950245 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.953249 (+ 3004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.953250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.953250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.953308 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.953502 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.953503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.953503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.953651 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:38.953734 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:38.953918 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 13:21:38.953995 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:21:38.955310 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.939827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.0.98.61:41157)
0925 13:21:38.939872 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:38.939874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:38.940147 (+ 273us) server_negotiation.cc:407] Connection header received
0925 13:21:38.940285 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:38.940286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:38.940312 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:38.940341 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:38.941368 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.942125 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.943562 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.943875 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.943916 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.946988 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:38.946990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:38.946991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:38.946999 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:38.950473 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.950950 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.950951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.950951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.951017 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.951289 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.951290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.951290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.951449 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:38.951515 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:38.952713 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0925 13:21:38.955222 (+ 2509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:21:38.955328 22793 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d [term 1 FOLLOWER]: Refusing update from remote peer 0c02ac5b9034426aa4d547de0814f41a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:38.955976 22852 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [LEADER]: Connected to new peer: Peer: permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 25 13:21:38 dist-test-slave-t8d9 krb5kdc[22588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:21:38.957392 22825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c02ac5b9034426aa4d547de0814f41a. Latest consensus state: current_term: 1 leader_uuid: "0c02ac5b9034426aa4d547de0814f41a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } } }
I20260925 13:21:38.957476 22825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:21:38.959456 22852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c02ac5b9034426aa4d547de0814f41a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } } }
I20260925 13:21:38.959547 22852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:38.959684 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.952172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41343 (local address 127.0.0.1:45156)
0925 13:21:38.952220 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:38.952231 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:38.952271 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:38.952740 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:38.952741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:38.952885 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:38.952926 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.952928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.954018 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.954018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.954759 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:38.954762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.955214 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.955215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:38.955268 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.955662 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:38.955666 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:38.956983 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:38.958677 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.958678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.958678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.958871 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.959060 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:38.959061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:38.959061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:38.959131 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:38.959404 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:38.959406 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:38.959450 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 13:21:38.959507 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 13:21:38.959599 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 13:21:38.960002 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:38.952269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45156 (local address 127.0.98.60:41343)
0925 13:21:38.952430 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:38.952432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:38.952439 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:21:38.952455 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:38.952456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:38.952465 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:38.952506 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:38.953155 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.953868 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.954837 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:38.955110 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:38.955164 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:38.957085 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:38.957086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:38.957087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:38.957094 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:38.958593 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.958943 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.958944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.958945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.958998 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:38.959195 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:38.959196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:38.959196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:38.959321 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:38.959377 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:38.959853 (+ 476us) server_negotiation.cc:299] Negotiation successful
0925 13:21:38.959922 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260925 13:21:38.960078 22825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c02ac5b9034426aa4d547de0814f41a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } } }
I20260925 13:21:38.960161 22825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:21:38.960541 22863 mvcc.cc:205] Tried to move back new op lower bound from 7333242875714387968 to 7333242875647827968. Current Snapshot: MvccSnapshot[applied={T|T < 7333242875714387968}]
I20260925 13:21:38.961102 22854 catalog_manager.cc:1347] Generated new cluster ID: 2e6e4b4b5d28433fa4378ecdeea03f81
I20260925 13:21:38.961148 22854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 13:21:38.961786 22744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b: Bootstrap starting.
I20260925 13:21:38.962409 22744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:38.962663 22744 log.cc:824] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:38.982122 22744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b: No bootstrap required, opened a new log
I20260925 13:21:38.984221 22744 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } }
I20260925 13:21:38.984375 22744 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5512f0b2e5f47bd96177089db95ab7b, State: Initialized, Role: FOLLOWER
I20260925 13:21:38.984445 22744 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } }
I20260925 13:21:38.984841 22718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c02ac5b9034426aa4d547de0814f41a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a5512f0b2e5f47bd96177089db95ab7b"
I20260925 13:21:38.984902 22717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:38.985500 22854 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:38.986040 22717 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b [term 1 FOLLOWER]: Refusing update from remote peer 0c02ac5b9034426aa4d547de0814f41a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:38.986418 22854 catalog_manager.cc:1504] Loading token signing keys...
I20260925 13:21:38.986372 22852 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:21:38.987126 22718 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c02ac5b9034426aa4d547de0814f41a in term 1.
I20260925 13:21:38.984812 22744 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:38.984779 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } } }
I20260925 13:21:38.989143 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:21:38.989208 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c02ac5b9034426aa4d547de0814f41a. Latest consensus state: current_term: 1 leader_uuid: "0c02ac5b9034426aa4d547de0814f41a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } } }
I20260925 13:21:38.989257 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:21:38.989286 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c02ac5b9034426aa4d547de0814f41a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } } }
I20260925 13:21:38.989326 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:21:38.992219 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c02ac5b9034426aa4d547de0814f41a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c02ac5b9034426aa4d547de0814f41a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40499 } } peers { permanent_uuid: "a5512f0b2e5f47bd96177089db95ab7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41157 } } peers { permanent_uuid: "01e4aa0c048f405fac3901eb7f9eb70d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41343 } } }
I20260925 13:21:38.992316 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:38.992987 22854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c02ac5b9034426aa4d547de0814f41a: Generated new TSK 0
I20260925 13:21:38.993300 22854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 13:21:38.993918 22870 mvcc.cc:205] Tried to move back new op lower bound from 7333242875827802112 to 7333242875647827968. Current Snapshot: MvccSnapshot[applied={T|T < 7333242875714387968}]
I20260925 13:21:38.994827 22883 catalog_manager.cc:1259] Loaded cluster ID: 2e6e4b4b5d28433fa4378ecdeea03f81
I20260925 13:21:38.994874 22883 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b: loading cluster ID for follower catalog manager: success
I20260925 13:21:38.995955 22883 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b: acquiring CA information for follower catalog manager: success
I20260925 13:21:38.996268 22883 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a5512f0b2e5f47bd96177089db95ab7b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 13:21:39.023700 22748 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:39.021997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.0.98.61:41157)
0925 13:21:39.022043 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.022045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.023012 (+ 967us) server_negotiation.cc:407] Connection header received
0925 13:21:39.023115 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.023116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.023128 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.023157 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:39.023613 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55260: BlockingRecv error: recv got EOF from 127.0.0.1:55260 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 13:21:39.024466 22673 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:39.022379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47736 (local address 127.0.98.62:40499)
0925 13:21:39.022429 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.022431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.023941 (+ 1510us) server_negotiation.cc:407] Connection header received
0925 13:21:39.024039 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.024040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.024051 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.024079 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:39.024393 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47736: BlockingRecv error: recv got EOF from 127.0.0.1:47736 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 13:21:39.025182 22823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:39.022755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.0.98.60:41343)
0925 13:21:39.022792 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.022794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.024704 (+ 1910us) server_negotiation.cc:407] Connection header received
0925 13:21:39.024796 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.024797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.024807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.024834 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:39.025120 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45164: BlockingRecv error: recv got EOF from 127.0.0.1:45164 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:21:39.030400 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:39.026857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.0.98.61:41157)
0925 13:21:39.026890 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.026892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.027082 (+ 190us) server_negotiation.cc:407] Connection header received
0925 13:21:39.027186 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.027187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.027231 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.027273 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:39.027459 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.028217 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.029379 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.029747 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.029787 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:39.030004 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:39.030142 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:39.030191 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:39.030297 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 13:21:39.030318 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 13:21:39.033874 22823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:39.033002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.0.98.60:41343)
0925 13:21:39.033060 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.033062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.033262 (+ 200us) server_negotiation.cc:407] Connection header received
0925 13:21:39.033371 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.033372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.033393 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.033421 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:39.033816 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45174: BlockingRecv error: recv got EOF from 127.0.0.1:45174 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:21:39.039615 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:39.036161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.0.98.61:41157)
0925 13:21:39.036199 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.036201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.036442 (+ 241us) server_negotiation.cc:407] Connection header received
0925 13:21:39.036531 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.036532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.036541 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.036579 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:39.036744 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.037485 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.038842 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.039130 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.039169 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:39.039350 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:39.039385 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:39.039409 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:39.039501 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 13:21:39.039521 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 13:21:39.042829 22823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:39.042060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45184 (local address 127.0.98.60:41343)
0925 13:21:39.042102 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.042104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.042281 (+ 177us) server_negotiation.cc:407] Connection header received
0925 13:21:39.042383 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.042384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.042395 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.042423 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:39.042769 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45184: BlockingRecv error: recv got EOF from 127.0.0.1:45184 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:21:39.049302 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:39.046068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55276 (local address 127.0.98.61:41157)
0925 13:21:39.046118 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.046120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.046364 (+ 244us) server_negotiation.cc:407] Connection header received
0925 13:21:39.046456 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.046457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.046467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.046495 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:39.046683 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.047417 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.048520 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.048827 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.048864 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:39.049046 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:39.049080 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:39.049106 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:39.049200 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 13:21:39.049219 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 13:21:39.052302 22823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:39.051583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45194 (local address 127.0.98.60:41343)
0925 13:21:39.051622 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.051624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.051795 (+ 171us) server_negotiation.cc:407] Connection header received
0925 13:21:39.051884 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.051884 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.051895 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.051922 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:39.052248 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45194: BlockingRecv error: recv got EOF from 127.0.0.1:45194 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:21:39.060887 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:39.057512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.0.98.61:41157)
0925 13:21:39.057617 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.057619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.057904 (+ 285us) server_negotiation.cc:407] Connection header received
0925 13:21:39.058001 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.058002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.058012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.058040 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:39.058211 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.058968 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.060100 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.060401 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.060438 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:39.060633 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:39.060667 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:39.060692 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:39.060786 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 13:21:39.060806 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260925 13:21:39.064116 22823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:39.063336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.0.98.60:41343)
0925 13:21:39.063389 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.063391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.063561 (+ 170us) server_negotiation.cc:407] Connection header received
0925 13:21:39.063668 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.063669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.063679 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.063708 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:39.064054 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45208: BlockingRecv error: recv got EOF from 127.0.0.1:45208 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 13:21:39.076956 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:39.073433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.0.98.61:41157)
0925 13:21:39.073480 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.073482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.073835 (+ 353us) server_negotiation.cc:407] Connection header received
0925 13:21:39.073940 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.073941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.073952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.073980 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:39.074285 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.075012 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.076138 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.076415 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.076461 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:39.076686 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:39.076723 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:39.076752 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:39.076850 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 13:21:39.076871 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 13:21:39.080238 22823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:39.079456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45220 (local address 127.0.98.60:41343)
0925 13:21:39.079498 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.079500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.079691 (+ 191us) server_negotiation.cc:407] Connection header received
0925 13:21:39.079777 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.079778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.079789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.079816 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:39.080177 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45220: BlockingRecv error: recv got EOF from 127.0.0.1:45220 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:21:39.100976 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:39.097732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.0.98.61:41157)
0925 13:21:39.097779 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.097781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.097876 (+ 95us) server_negotiation.cc:407] Connection header received
0925 13:21:39.097959 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.097961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.097972 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.098028 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:39.098176 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.098962 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.100153 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.100428 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.100514 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:39.100712 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:39.100748 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:39.100799 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:39.100874 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 13:21:39.100894 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 13:21:39.103703 22823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:39.103033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.0.98.60:41343)
0925 13:21:39.103075 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.103077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.103183 (+ 106us) server_negotiation.cc:407] Connection header received
0925 13:21:39.103244 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.103245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.103258 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.103315 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:39.103644 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45234: BlockingRecv error: recv got EOF from 127.0.0.1:45234 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 13:21:39.139366 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:39.136784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.0.98.61:41157)
0925 13:21:39.136857 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.136859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.136909 (+ 50us) server_negotiation.cc:407] Connection header received
0925 13:21:39.136970 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.136971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.136981 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.137057 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:39.137177 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.137964 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.138696 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.138967 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.139039 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:39.139144 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:39.139181 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:39.139228 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:39.139265 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 13:21:39.139282 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 13:21:39.141510 22823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:39.141007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.0.98.60:41343)
0925 13:21:39.141054 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.141056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.141064 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:21:39.141087 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.141088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.141099 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.141144 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:39.141453 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45248: BlockingRecv error: recv got EOF from 127.0.0.1:45248 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:21:39.208796 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:39.206595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.0.98.61:41157)
0925 13:21:39.206645 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.206647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.206751 (+ 104us) server_negotiation.cc:407] Connection header received
0925 13:21:39.206818 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.206819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.206830 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.206874 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:39.206998 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.207483 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.208211 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.208405 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.208457 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:39.208581 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:39.208611 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:39.208648 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:39.208697 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 13:21:39.208716 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 13:21:39.210942 22823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:39.210378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45254 (local address 127.0.98.60:41343)
0925 13:21:39.210444 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.210446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.210480 (+ 34us) server_negotiation.cc:407] Connection header received
0925 13:21:39.210551 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.210552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.210562 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.210602 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:39.210881 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45254: BlockingRecv error: recv got EOF from 127.0.0.1:45254 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 13:21:39.342312 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:39.340038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.0.98.61:41157)
0925 13:21:39.340116 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.340119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.340196 (+ 77us) server_negotiation.cc:407] Connection header received
0925 13:21:39.340268 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.340270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.340280 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.340330 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:39.340464 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.340949 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.341706 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.341892 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.341953 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:39.342079 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:39.342111 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:39.342151 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:39.342209 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 13:21:39.342229 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 13:21:39.344415 22823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:39.343913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45262 (local address 127.0.98.60:41343)
0925 13:21:39.344011 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.344013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.344040 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:21:39.344061 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.344062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.344073 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.344109 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:39.344359 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45262: BlockingRecv error: recv got EOF from 127.0.0.1:45262 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 13:21:39.603895 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:39.601607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.0.98.61:41157)
0925 13:21:39.601668 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.601669 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.601680 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:21:39.601750 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.601751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.601759 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.601807 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:39.601949 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.602420 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.603204 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:39.603399 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:39.603464 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:39.603636 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:39.603671 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:39.603718 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:39.603813 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 13:21:39.603831 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 13:21:39.606133 22823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:39.605586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.0.98.60:41343)
0925 13:21:39.605645 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:39.605647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:39.605655 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:21:39.605706 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:39.605707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:39.605716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:39.605760 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:39.606071 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45278: BlockingRecv error: recv got EOF from 127.0.0.1:45278 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 13:21:39.902593 22836 catalog_manager.cc:1259] Loaded cluster ID: 2e6e4b4b5d28433fa4378ecdeea03f81
I20260925 13:21:39.902673 22836 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d: loading cluster ID for follower catalog manager: success
I20260925 13:21:39.903596 22836 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d: acquiring CA information for follower catalog manager: success
I20260925 13:21:39.903906 22836 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 01e4aa0c048f405fac3901eb7f9eb70d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 13:21:40.121486 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:40.119292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55368 (local address 127.0.98.61:41157)
0925 13:21:40.119422 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:40.119424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:40.119437 (+ 13us) server_negotiation.cc:407] Connection header received
0925 13:21:40.119518 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:40.119519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:40.119530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:40.119581 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:40.119695 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:40.120222 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:40.120914 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:40.121114 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:40.121160 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:40.121271 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:40.121300 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:40.121333 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:40.121372 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 13:21:40.121388 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260925 13:21:40.125499 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:40.123088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45294 (local address 127.0.98.60:41343)
0925 13:21:40.123191 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:40.123193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:40.123205 (+ 12us) server_negotiation.cc:407] Connection header received
0925 13:21:40.123272 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:40.123273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:40.123344 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:40.123410 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:40.123526 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:40.124031 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:40.124741 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:40.124946 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:40.125000 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:40.125163 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:40.125279 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:40.125347 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:40.125385 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 13:21:40.125407 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":48,"threads_started":1}
I20260925 13:21:40.130014 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:40.127105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.0.98.60:41343)
0925 13:21:40.127252 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:40.127253 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:40.127261 (+ 8us) server_negotiation.cc:407] Connection header received
0925 13:21:40.127305 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:40.127306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:40.127318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:40.127345 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:40.127615 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:40.128363 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:40.129323 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:40.129515 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:40.129598 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:40.129745 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:40.129779 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:40.129824 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:40.129894 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 13:21:40.129919 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260925 13:21:40.130077 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:40.127116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.0.98.61:41157)
0925 13:21:40.127160 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:40.127163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:40.127426 (+ 263us) server_negotiation.cc:407] Connection header received
0925 13:21:40.127504 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:40.127505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:40.127519 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:40.127563 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:40.127683 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:40.128179 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:40.129236 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:40.129523 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:40.129604 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:40.129798 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:40.129833 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:40.129881 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:40.129969 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 13:21:40.129998 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:21:40.130486 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:40.127165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47740 (local address 127.0.98.62:40499)
0925 13:21:40.127311 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:40.127313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:40.127382 (+ 69us) server_negotiation.cc:407] Connection header received
0925 13:21:40.127469 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:40.127470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:40.127528 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:40.127613 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:40.127731 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:40.128499 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:40.129336 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:40.129671 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:40.129731 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:40.130151 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:40.130273 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:40.130331 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:40.130377 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 13:21:40.130401 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":50,"threads_started":1}
I20260925 13:21:40.141243 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 22601
I20260925 13:21:40.146421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 22676
I20260925 13:21:40.150985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 22751
2026-09-25T13:21:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2736 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 13:21:40 dist-test-slave-t8d9 krb5kdc[23020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 13:21:40 dist-test-slave-t8d9 krb5kdc[23020](info): set up 2 sockets
Sep 25 13:21:40 dist-test-slave-t8d9 krb5kdc[23020](info): commencing operation
krb5kdc: starting...
W20260925 13:21:41.101643 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/test-user.keytab.
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T13:21:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T13:21:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 13:21:41.213616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42793
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42793,127.0.98.61:45039,127.0.98.60:44871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 13:21:41.293869 23033 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 13:21:41.294117 23033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:41.294133 23033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:41.294168 23033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:41.295451 23033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:41.295506 23033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:41.295518 23033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:41.295531 23033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:41.295542 23033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:41.297139 23033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42793,127.0.98.61:45039,127.0.98.60:44871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:41.297399 23033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:41.297667 23033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:41.302402 23038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:41.302461 23039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:41.302412 23041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:41.302644 23033 server_base.cc:1034] running on GCE node
I20260925 13:21:41.302872 23033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:41.303185 23033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:41.304356 23033 hybrid_clock.cc:648] HybridClock initialized: now 1790342501304320 us; error 50 us; skew 500 ppm
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:41.306842 23033 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 13:21:41.307312 23033 webserver.cc:460] Webserver started at http://127.0.98.62:42493/ using document root <none> and password file <none>
I20260925 13:21:41.307605 23033 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:41.307667 23033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:41.307796 23033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:41.308943 23033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/data/instance:
uuid: "729697013d004b259b010cd5c17ed091"
format_stamp: "Formatted at 2026-09-25 13:21:41 on dist-test-slave-t8d9"
server_key: "318589f32aa6d65668235a4f6a8eba3e"
server_key_iv: "748bec3a8e6b368c15729bffa622876a"
server_key_version: "encryptionkey@0"
I20260925 13:21:41.309268 23033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/wal/instance:
uuid: "729697013d004b259b010cd5c17ed091"
format_stamp: "Formatted at 2026-09-25 13:21:41 on dist-test-slave-t8d9"
server_key: "318589f32aa6d65668235a4f6a8eba3e"
server_key_iv: "748bec3a8e6b368c15729bffa622876a"
server_key_version: "encryptionkey@0"
I20260925 13:21:41.310670 23033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 13:21:41.311539 23048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:41.311751 23033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:21:41.311846 23033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/wal
uuid: "729697013d004b259b010cd5c17ed091"
format_stamp: "Formatted at 2026-09-25 13:21:41 on dist-test-slave-t8d9"
server_key: "318589f32aa6d65668235a4f6a8eba3e"
server_key_iv: "748bec3a8e6b368c15729bffa622876a"
server_key_version: "encryptionkey@0"
I20260925 13:21:41.311918 23033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:41.329861 23033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:41.331923 23033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:41.332043 23033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:41.335875 23033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42793
I20260925 13:21:41.335908 23100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42793 every 8 connection(s)
I20260925 13:21:41.336249 23033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/data/info.pb
I20260925 13:21:41.337255 23101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:41.338429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 23033
I20260925 13:21:41.338513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-0/wal/instance
I20260925 13:21:41.338701 392 external_mini_cluster.cc:1428] Setting key 1bafa3d9008cfc7c4209706540a49014
I20260925 13:21:41.338953 23101 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260925 13:21:41.349673 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.339670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42793 (local address 127.0.0.1:46010)
0925 13:21:41.339846 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.339893 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:41.340508 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:41.340802 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:41.340805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:41.341135 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:41.341297 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.341302 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.342878 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.342879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.343587 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.343590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.344287 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.344289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.344346 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.344861 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:41.344869 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:41.346365 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:41.348111 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.348118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.348121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.348340 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.348608 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.348609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.348609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.348696 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.349088 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:41.349092 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:41.349164 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 13:21:41.349242 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 13:21:41.349461 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":72,"thread_start_us":83,"threads_started":1}
I20260925 13:21:41.349670 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.339736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46010 (local address 127.0.98.62:42793)
0925 13:21:41.339950 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.339955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.339991 (+ 36us) server_negotiation.cc:407] Connection header received
0925 13:21:41.340669 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.340671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.340689 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.340766 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.341413 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.342744 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.343679 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.344178 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.344302 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.346449 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:41.346456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:41.346457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:41.346463 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:41.348003 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.348441 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.348445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.348445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.348548 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.348763 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.348765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.348765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.349016 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:41.349101 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:41.349274 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 13:21:41.349390 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
I20260925 13:21:41.350760 23101 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:21:41.351114 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.350992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45039 (local address 127.0.0.1:40558)
0925 13:21:41.351038 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.351088 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 13:21:41.351279 23050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45039: connect: Connection refused (error 111)
I20260925 13:21:41.351696 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.351602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45039 (local address 127.0.0.1:40574)
0925 13:21:41.351646 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.351674 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 13:21:41.351864 23101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45039: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45039: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 13:21:41.361735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45039
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42793,127.0.98.61:45039,127.0.98.60:44871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:21:41.401120 23101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } attempt: 1
I20260925 13:21:41.401700 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.401552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45039 (local address 127.0.0.1:40584)
0925 13:21:41.401611 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.401664 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 13:21:41.402036 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.401949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45039 (local address 127.0.0.1:40596)
0925 13:21:41.401987 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.402015 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 13:21:41.402145 23101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45039: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45039: connect: Connection refused (error 111)
W20260925 13:21:41.442986 23108 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 13:21:41.443215 23108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:41.443233 23108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:41.443252 23108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:41.444428 23108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:41.444476 23108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:41.444489 23108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:41.444504 23108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:41.444514 23108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:41.446107 23108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42793,127.0.98.61:45039,127.0.98.60:44871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:41.446346 23108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:41.446569 23108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:41.451256 23114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:41.451243 23116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:41.451247 23113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:41.451573 23108 server_base.cc:1034] running on GCE node
I20260925 13:21:41.451817 23108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:41.452097 23108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:41.453241 23108 hybrid_clock.cc:648] HybridClock initialized: now 1790342501453228 us; error 28 us; skew 500 ppm
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:41.455657 23108 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 13:21:41.456115 23108 webserver.cc:460] Webserver started at http://127.0.98.61:44317/ using document root <none> and password file <none>
I20260925 13:21:41.456310 23108 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:41.456349 23108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:41.456442 23108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:41.457506 23108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/data/instance:
uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5"
format_stamp: "Formatted at 2026-09-25 13:21:41 on dist-test-slave-t8d9"
server_key: "afa6239d55cfe0a22ede617e702ea58a"
server_key_iv: "f9db9a5fa7c787d52d0733ca66b6d6c9"
server_key_version: "encryptionkey@0"
I20260925 13:21:41.457845 23108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/wal/instance:
uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5"
format_stamp: "Formatted at 2026-09-25 13:21:41 on dist-test-slave-t8d9"
server_key: "afa6239d55cfe0a22ede617e702ea58a"
server_key_iv: "f9db9a5fa7c787d52d0733ca66b6d6c9"
server_key_version: "encryptionkey@0"
I20260925 13:21:41.459050 23108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 13:21:41.459735 23123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:41.459877 23108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:41.459959 23108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/wal
uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5"
format_stamp: "Formatted at 2026-09-25 13:21:41 on dist-test-slave-t8d9"
server_key: "afa6239d55cfe0a22ede617e702ea58a"
server_key_iv: "f9db9a5fa7c787d52d0733ca66b6d6c9"
server_key_version: "encryptionkey@0"
I20260925 13:21:41.460035 23108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:41.470331 23101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } attempt: 2
I20260925 13:21:41.470842 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.470676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45039 (local address 127.0.0.1:40606)
0925 13:21:41.470759 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.470808 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 13:21:41.471129 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.471060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45039 (local address 127.0.0.1:40608)
0925 13:21:41.471086 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.471110 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260925 13:21:41.471223 23101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45039: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45039: connect: Connection refused (error 111)
I20260925 13:21:41.475782 23108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:41.478003 23108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:41.478145 23108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:41.481930 23108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45039
I20260925 13:21:41.481964 23175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45039 every 8 connection(s)
I20260925 13:21:41.482317 23108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/data/info.pb
I20260925 13:21:41.483260 23176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:41.484822 23176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:21:41.486274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 23108
I20260925 13:21:41.486377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-1/wal/instance
I20260925 13:21:41.486554 392 external_mini_cluster.cc:1428] Setting key 858c09b77fe5ca8804f44b545a048fa0
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260925 13:21:41.494410 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.485563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.0.98.62:42793)
0925 13:21:41.485607 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.485608 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.485815 (+ 207us) server_negotiation.cc:407] Connection header received
0925 13:21:41.486292 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.486293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.486310 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.486390 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.487103 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.487775 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.489197 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.489404 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.489464 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.491633 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:41.491635 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:41.491635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:41.491644 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:41.493053 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.493448 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.493449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.493449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.493504 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.493781 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.493782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.493782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.493894 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:41.493949 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:41.494227 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 13:21:41.494315 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:21:41.494611 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.485483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42793 (local address 127.0.0.1:46022)
0925 13:21:41.485688 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.485732 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:41.486196 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:41.486479 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:41.486483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:41.486769 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:41.487005 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.487011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.487908 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.487909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.489114 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.489117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.489498 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.489499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.489658 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.490166 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:41.490175 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:41.491536 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:41.493147 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.493155 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.493158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.493381 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.493618 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.493619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.493619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.493724 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.493950 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:41.493957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:41.494110 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 13:21:41.494202 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 13:21:41.494382 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":107,"threads_started":1}
I20260925 13:21:41.495157 23176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:21:41.502413 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.495396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45039 (local address 127.0.0.1:40618)
0925 13:21:41.495463 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.495475 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:41.495521 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:41.495815 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:41.495816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:41.495926 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:41.495992 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.495993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.496830 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.496831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.497424 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.497426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.498102 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.498103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.498182 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.498568 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:41.498573 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:41.499822 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:41.501403 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.501403 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.501404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.501607 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.501778 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.501779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.501779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.501833 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.502086 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:41.502087 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:41.502126 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 13:21:41.502184 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 13:21:41.502293 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 13:21:41.502435 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.495489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.0.98.61:45039)
0925 13:21:41.495665 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.495670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.495694 (+ 24us) server_negotiation.cc:407] Connection header received
0925 13:21:41.495714 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.495716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.495729 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.495793 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.496062 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.496755 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.497505 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.498024 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.498094 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.499906 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:41.499916 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:41.499917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:41.499923 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:41.501307 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.501673 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.501677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.501677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.501726 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.501880 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.501880 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.501880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.502027 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:41.502099 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:41.502213 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 13:21:41.502345 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260925 13:21:41.503026 23176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:21:41.503335 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.503234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44871 (local address 127.0.0.1:44596)
0925 13:21:41.503267 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.503313 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260925 13:21:41.503463 23125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
I20260925 13:21:41.503830 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.503768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44871 (local address 127.0.0.1:44608)
0925 13:21:41.503791 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.503814 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260925 13:21:41.503969 23176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44871: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 13:21:41.510365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bpD2/build/release/bin/kudu
/tmp/dist-test-taskC7bpD2/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44871
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42793,127.0.98.61:45039,127.0.98.60:44871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 13:21:41.553323 23176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } attempt: 1
I20260925 13:21:41.553867 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.553727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44871 (local address 127.0.0.1:44620)
0925 13:21:41.553786 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.553829 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 13:21:41.554239 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.554144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44871 (local address 127.0.0.1:44636)
0925 13:21:41.554191 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.554216 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 13:21:41.554324 23176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44871: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
I20260925 13:21:41.562386 23101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } attempt: 3
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:21:41.569515 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.562711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45039 (local address 127.0.0.1:40630)
0925 13:21:41.562773 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.562784 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:41.562831 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:41.562950 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:41.562951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:41.563128 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:41.563191 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.563193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.563884 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.563886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.564432 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.564436 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.564783 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.564784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.564859 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.565321 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:41.565326 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:41.566747 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:41.568549 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.568550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.568551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.568723 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.568899 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.568900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.568900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.568957 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.569186 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:41.569187 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:41.569238 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 13:21:41.569296 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 13:21:41.569402 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 13:21:41.569509 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.562775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.0.98.61:45039)
0925 13:21:41.562833 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.562835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.562862 (+ 27us) server_negotiation.cc:407] Connection header received
0925 13:21:41.562887 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.562888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.562899 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.562940 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.563262 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.563802 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.564509 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.564714 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.564769 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.566843 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:41.566846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:41.566847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:41.566854 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:41.568435 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.568788 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.568790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.568790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.568843 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.569013 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.569014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.569014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.569129 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:41.569183 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:41.569323 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 13:21:41.569402 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:21:41.570109 23101 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:21:41.570432 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.570335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44871 (local address 127.0.0.1:44648)
0925 13:21:41.570376 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.570405 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 13:21:41.570745 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.570662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44871 (local address 127.0.0.1:44664)
0925 13:21:41.570707 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.570730 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 13:21:41.570842 23101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44871: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
W20260925 13:21:41.591337 23183 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 13:21:41.591565 23183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 13:21:41.591591 23183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 13:21:41.591612 23183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 13:21:41.592774 23183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 13:21:41.592813 23183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 13:21:41.592825 23183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 13:21:41.592836 23183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 13:21:41.592849 23183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 13:21:41.594430 23183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42793,127.0.98.61:45039,127.0.98.60:44871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:18:27 UTC on ccycloud.quasar-mgemhg.root.comops.site
build id 4672476
I20260925 13:21:41.594683 23183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 13:21:41.594901 23183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 13:21:41.599472 23189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:41.599478 23188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 13:21:41.599483 23191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 13:21:41.599711 23183 server_base.cc:1034] running on GCE node
I20260925 13:21:41.599982 23183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 13:21:41.600252 23183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 13:21:41.601408 23183 hybrid_clock.cc:648] HybridClock initialized: now 1790342501601389 us; error 26 us; skew 500 ppm
I20260925 13:21:41.601967 23101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } attempt: 1
I20260925 13:21:41.602357 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.602208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44871 (local address 127.0.0.1:44666)
0925 13:21:41.602276 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.602320 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 13:21:41.602675 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.602601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44871 (local address 127.0.0.1:44682)
0925 13:21:41.602635 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.602658 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 13:21:41.602767 23101 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44871: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 13:21:41.604082 23183 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 13:21:41.604589 23183 webserver.cc:460] Webserver started at http://127.0.98.60:45089/ using document root <none> and password file <none>
I20260925 13:21:41.604806 23183 fs_manager.cc:359] Metadata directory not provided
I20260925 13:21:41.604854 23183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 13:21:41.604980 23183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 13:21:41.606112 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/data/instance:
uuid: "bfbd89659e614029aec03ae01c529610"
format_stamp: "Formatted at 2026-09-25 13:21:41 on dist-test-slave-t8d9"
server_key: "6a2febb5b77a65827ad69d9f129b5754"
server_key_iv: "b473b48ffed2d97541e0452004ab6ae8"
server_key_version: "encryptionkey@0"
I20260925 13:21:41.606459 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/wal/instance:
uuid: "bfbd89659e614029aec03ae01c529610"
format_stamp: "Formatted at 2026-09-25 13:21:41 on dist-test-slave-t8d9"
server_key: "6a2febb5b77a65827ad69d9f129b5754"
server_key_iv: "b473b48ffed2d97541e0452004ab6ae8"
server_key_version: "encryptionkey@0"
I20260925 13:21:41.607733 23183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 13:21:41.608482 23198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 13:21:41.608654 23183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 13:21:41.608733 23183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/wal
uuid: "bfbd89659e614029aec03ae01c529610"
format_stamp: "Formatted at 2026-09-25 13:21:41 on dist-test-slave-t8d9"
server_key: "6a2febb5b77a65827ad69d9f129b5754"
server_key_iv: "b473b48ffed2d97541e0452004ab6ae8"
server_key_version: "encryptionkey@0"
I20260925 13:21:41.608803 23183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 13:21:41.622481 23176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } attempt: 2
I20260925 13:21:41.622908 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.622787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44871 (local address 127.0.0.1:44690)
0925 13:21:41.622843 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.622879 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 13:21:41.623186 23183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 13:21:41.623208 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.623129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44871 (local address 127.0.0.1:44696)
0925 13:21:41.623166 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.623189 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 13:21:41.623291 23176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44871: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44871: connect: Connection refused (error 111)
I20260925 13:21:41.624994 23183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 13:21:41.625124 23183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 13:21:41.628873 23183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44871
I20260925 13:21:41.628916 23250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44871 every 8 connection(s)
I20260925 13:21:41.629238 23183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/data/info.pb
I20260925 13:21:41.630203 23251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 13:21:41.631807 23251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:21:41.635046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bpD2/build/release/bin/kudu as pid 23183
I20260925 13:21:41.635160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bpD2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790342239834770-392-0/minicluster-data/master-2/wal/instance
I20260925 13:21:41.635339 392 external_mini_cluster.cc:1428] Setting key 4005c19f9d504fa850fcb7b538b17d7e
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:21:41.641384 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.632574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46024 (local address 127.0.98.62:42793)
0925 13:21:41.632639 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.632641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.632830 (+ 189us) server_negotiation.cc:407] Connection header received
0925 13:21:41.633298 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.633300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.633309 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.633362 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.634057 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.634538 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.635684 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.635957 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.636023 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.638401 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:41.638403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:41.638403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:41.638413 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:41.639844 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.640313 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.640314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.640314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.640382 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.640666 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.640667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.640667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.640806 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:41.640860 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:41.641201 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 13:21:41.641288 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 13:21:41.641499 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.632530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42793 (local address 127.0.0.1:46024)
0925 13:21:41.632713 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.632752 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:41.633214 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:41.633432 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:41.633436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:41.633756 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:41.633963 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.633969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.634625 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.634625 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.635588 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.635591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.636045 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.636046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.636190 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.636728 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:41.636737 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:41.638315 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:41.639921 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.639928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.639930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.640224 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.640480 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.640481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.640481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.640596 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.640870 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:41.640877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:41.641049 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 13:21:41.641162 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 13:21:41.641302 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260925 13:21:41.642110 23251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:21:41.644001 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.635734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.0.98.60:44871)
0925 13:21:41.635875 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.635880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.635923 (+ 43us) server_negotiation.cc:407] Connection header received
0925 13:21:41.636029 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.636032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.636056 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.636146 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.636788 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.637723 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.638521 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.638812 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.638876 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.641185 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:41.641193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:41.641194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:41.641201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:41.642927 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.643237 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.643241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.643241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.643295 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.643461 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.643462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.643462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.643609 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:41.643730 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:41.643827 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 13:21:41.643902 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:21:41.648841 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.642364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45039 (local address 127.0.0.1:40638)
0925 13:21:41.642473 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.642483 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:41.642526 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:41.642629 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:41.642630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:41.642831 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:41.642863 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.642864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.643448 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.643448 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.643943 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.643945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.644446 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.644447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.644504 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.644865 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:41.644869 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:41.646274 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:41.647869 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.647869 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.647870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.648042 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.648230 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.648230 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.648231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.648274 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.648546 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:41.648548 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:41.648589 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 13:21:41.648641 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 13:21:41.648747 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 13:21:41.648840 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.642430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40638 (local address 127.0.98.61:45039)
0925 13:21:41.642478 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.642480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.642512 (+ 32us) server_negotiation.cc:407] Connection header received
0925 13:21:41.642573 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.642574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.642584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.642628 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.642929 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.643383 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.644115 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.644381 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.644448 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.646344 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:41.646345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:41.646345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:41.646354 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:41.647796 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.648109 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.648110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.648110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.648169 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.648319 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.648320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.648320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.648455 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:41.648519 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:41.648682 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 13:21:41.648751 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 13:21:41.649176 23251 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } has no permanent_uuid. Determining permanent_uuid...
I20260925 13:21:41.651723 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.644697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.0.98.62:42793)
0925 13:21:41.644743 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.644745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.644754 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:21:41.644788 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.644789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.644798 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.644844 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.645367 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.646101 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.646899 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.647146 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.647220 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.649367 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:41.649369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:41.649370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:41.649377 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:41.650768 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.651061 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.651061 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.651062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.651116 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.651259 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.651260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.651260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.651370 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:41.651488 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:41.651561 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 13:21:41.651638 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:21:41.655334 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.649407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44871 (local address 127.0.0.1:44700)
0925 13:21:41.649493 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.649502 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:41.649572 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:41.649695 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:41.649696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:41.649829 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:41.649865 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.649867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.650574 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.650575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.651212 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.651214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.651588 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.651589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.651642 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.651970 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:41.651974 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:41.653129 (+ 1155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:41.654499 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.654500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.654501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.654640 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.654791 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.654791 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.654791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.654840 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.655039 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:41.655040 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:41.655082 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 13:21:41.655138 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 13:21:41.655227 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 13:21:41.655378 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.649460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.0.98.60:44871)
0925 13:21:41.649510 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.649512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.649559 (+ 47us) server_negotiation.cc:407] Connection header received
0925 13:21:41.649621 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.649622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.649632 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.649693 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.649930 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.650499 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.651277 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.651523 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.651578 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.653200 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:41.653201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:41.653201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:41.653206 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:41.654416 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.654696 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.654697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.654697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.654742 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.654887 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.654888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.654888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.654983 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:41.655034 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:41.655168 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 13:21:41.655271 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":32}
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 13:21:41.656973 23251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610: Bootstrap starting.
I20260925 13:21:41.657904 23251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:41.658175 23251 log.cc:824] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:41.659003 23251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610: No bootstrap required, opened a new log
I20260925 13:21:41.659233 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.652394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.0.98.61:45039)
0925 13:21:41.652438 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.652440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.652449 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:21:41.652471 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.652472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.652482 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.652541 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.652979 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.653510 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.654429 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.654630 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.654684 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.656860 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:41.656862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:41.656862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:41.656868 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:41.658195 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.658548 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.658549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.658550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.658612 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.658797 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.658798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.658798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.658903 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:41.659039 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:41.659098 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 13:21:41.659168 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:21:41.660478 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 13:21:41.660684 23251 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } }
I20260925 13:21:41.660807 23251 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfbd89659e614029aec03ae01c529610, State: Initialized, Role: FOLLOWER
I20260925 13:21:41.660892 23251 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } }
I20260925 13:21:41.661163 23257 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } } }
I20260925 13:21:41.661222 23251 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:41.661244 23257 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:21:41.661444 23075 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "729697013d004b259b010cd5c17ed091"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46036
W20260925 13:21:41.663166 23268 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:21:41.663218 23268 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:21:41.663379 23183 master_runner.cc:186] Error getting master registration for 127.0.98.62:42793: OK, instance_id { permanent_uuid: "729697013d004b259b010cd5c17ed091" instance_seqno: 1790342501334795 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 13:21:41.666174 23183 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 13:21:41.677918 23101 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } attempt: 2
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:21:41.685868 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.678343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44710 (local address 127.0.98.60:44871)
0925 13:21:41.678455 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.678458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.678483 (+ 25us) server_negotiation.cc:407] Connection header received
0925 13:21:41.678521 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.678522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.678532 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.678589 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.678934 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.679690 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.680677 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.680984 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.681049 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.683115 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:41.683117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:41.683117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:41.683125 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:41.684650 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.685023 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.685024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.685025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.685077 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.685252 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.685253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.685253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.685443 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:41.685512 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:41.685686 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 13:21:41.685768 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 13:21:41.685870 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.678288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44871 (local address 127.0.0.1:44710)
0925 13:21:41.678363 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.678374 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:41.678416 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:41.678613 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:41.678614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:41.678806 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:41.678863 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.678865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.679774 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.679775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.680596 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.680600 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.681054 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.681055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.681116 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.681663 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:41.681669 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:41.683017 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:41.684732 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.684733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.684734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.684958 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.685135 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.685136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.685136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.685199 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.685504 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:41.685505 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:41.685583 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 13:21:41.685662 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 13:21:41.685765 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 13:21:41.687687 23101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091: Bootstrap starting.
I20260925 13:21:41.688355 23101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:41.688627 23101 log.cc:824] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:41.689337 23101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091: No bootstrap required, opened a new log
I20260925 13:21:41.690392 23101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } }
I20260925 13:21:41.690512 23101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729697013d004b259b010cd5c17ed091, State: Initialized, Role: FOLLOWER
I20260925 13:21:41.690591 23101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } }
I20260925 13:21:41.690834 23271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } } }
I20260925 13:21:41.690891 23101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:41.690915 23271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:21:41.691023 23075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 13:21:41.691138 23075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:41.691952 23075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } }
I20260925 13:21:41.692339 23075 leader_election.cc:288] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [CANDIDATE]: Term 1 election: Requested vote from peers 6e3895258b1f42f7a4adcc0d3be8f4f5 (127.0.98.61:45039), bfbd89659e614029aec03ae01c529610 (127.0.98.60:44871)
I20260925 13:21:41.692611 23225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "729697013d004b259b010cd5c17ed091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bfbd89659e614029aec03ae01c529610"
I20260925 13:21:41.692734 23225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610 [term 0 FOLLOWER]: Advancing to term 1
W20260925 13:21:41.693357 23282 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:21:41.693502 23282 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:21:41.693692 23225 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 729697013d004b259b010cd5c17ed091 in term 1.
I20260925 13:21:41.694074 23050 leader_election.cc:302] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 729697013d004b259b010cd5c17ed091, bfbd89659e614029aec03ae01c529610; no voters:
I20260925 13:21:41.694200 23271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 13:21:41.694327 23271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [term 1 LEADER]: Becoming Leader. State: Replica: 729697013d004b259b010cd5c17ed091, State: Running, Role: LEADER
I20260925 13:21:41.694415 23271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } }
I20260925 13:21:41.694949 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 729697013d004b259b010cd5c17ed091. Latest consensus state: current_term: 1 leader_uuid: "729697013d004b259b010cd5c17ed091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } } }
I20260925 13:21:41.695101 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:41.695303 23290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 13:21:41.695488 23290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 13:21:41.694725 23033 master_runner.cc:186] Error getting master registration for 127.0.98.61:45039: OK, instance_id { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" instance_seqno: 1790342501480830 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 13:21:41.696631 23033 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 13:21:41.698020 23225 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610 [term 1 FOLLOWER]: Refusing update from remote peer 729697013d004b259b010cd5c17ed091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:41.698400 23288 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:21:41.699790 23257 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 729697013d004b259b010cd5c17ed091. Latest consensus state: current_term: 1 leader_uuid: "729697013d004b259b010cd5c17ed091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } } }
I20260925 13:21:41.699873 23257 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:21:41.706029 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.696923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.0.98.61:45039)
0925 13:21:41.696967 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.696969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.697219 (+ 250us) server_negotiation.cc:407] Connection header received
0925 13:21:41.697318 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.697319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.697332 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.697362 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.698164 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.698906 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.699904 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.700178 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.700217 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.701774 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:41.701775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:41.701776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:41.701783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:41.703349 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.703879 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.703881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.703881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.703957 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.704912 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.704914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.704914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.705060 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:41.705609 (+ 549us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:41.705854 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 13:21:41.705937 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 13:21:41.707749 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "729697013d004b259b010cd5c17ed091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } } }
I20260925 13:21:41.707842 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:41.707911 23295 mvcc.cc:205] Tried to move back new op lower bound from 7333242886952243200 to 7333242886941069312. Current Snapshot: MvccSnapshot[applied={T|T < 7333242886952243200}]
I20260925 13:21:41.708001 23257 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "729697013d004b259b010cd5c17ed091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } } }
I20260925 13:21:41.708072 23257 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:21:41.708431 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.697263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.0.98.60:44871)
0925 13:21:41.697300 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.697302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.697816 (+ 514us) server_negotiation.cc:407] Connection header received
0925 13:21:41.697912 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.697914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.697924 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.697952 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.701618 (+ 3666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.702183 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.702972 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.703242 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.703305 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.704768 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:41.704769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:41.704769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:41.704776 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:41.706508 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.706964 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.706965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.706966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.707037 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.707237 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.707238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.707238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.707369 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:41.707449 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:41.708260 (+ 811us) server_negotiation.cc:299] Negotiation successful
0925 13:21:41.708339 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 13:21:41.708446 23291 mvcc.cc:205] Tried to move back new op lower bound from 7333242886952243200 to 7333242886941069312. Current Snapshot: MvccSnapshot[applied={T|T < 7333242886952243200}]
I20260925 13:21:41.714421 23176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } attempt: 3
I20260925 13:21:41.714679 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.697563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.0.98.62:42793)
0925 13:21:41.697599 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.697601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.705687 (+ 8086us) server_negotiation.cc:407] Connection header received
0925 13:21:41.705797 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.705798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.705809 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.705838 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.707272 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.707980 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.708894 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.709184 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.709220 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.711436 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:41.711438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:41.711438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:41.711473 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:41.713071 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.713697 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.713699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.713699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.713759 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.714025 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.714026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.714026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.714164 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:41.714281 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:41.714491 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 13:21:41.714583 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:21:41.718408 23290 catalog_manager.cc:1347] Generated new cluster ID: bdeb46d1ee434e159007c5960eaee27a
I20260925 13:21:41.718465 23290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 13:21:41 dist-test-slave-t8d9 krb5kdc[23020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790342501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 13:21:41.722950 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.714716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44871 (local address 127.0.0.1:44738)
0925 13:21:41.714883 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 13:21:41.714894 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 13:21:41.714935 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 13:21:41.715178 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 13:21:41.715179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 13:21:41.715338 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 13:21:41.715381 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.715383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.716304 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.716305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.717125 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 13:21:41.717128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.717657 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.717659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 13:21:41.717715 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.718189 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 13:21:41.718195 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 13:21:41.719571 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 13:21:41.721652 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.721653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.721654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.721864 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.722305 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 13:21:41.722307 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 13:21:41.722307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 13:21:41.722377 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 13:21:41.722662 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 13:21:41.722663 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 13:21:41.722710 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 13:21:41.722756 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 13:21:41.722858 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260925 13:21:41.723371 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:41.714870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44738 (local address 127.0.98.60:44871)
0925 13:21:41.715051 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.715052 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.715061 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:21:41.715099 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.715099 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.715109 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.715170 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.715454 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.716187 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.717206 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:41.717519 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:41.717643 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:41.719649 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 13:21:41.719651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 13:21:41.719651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 13:21:41.719658 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 13:21:41.721510 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.721928 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.721929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.721929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.722219 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 13:21:41.722446 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 13:21:41.722447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 13:21:41.722447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 13:21:41.722590 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 13:21:41.723169 (+ 579us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:41.723214 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 13:21:41.723287 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260925 13:21:41.726035 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5: Bootstrap starting.
I20260925 13:21:41.726688 23176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5: Neither blocks nor log segments found. Creating new log.
I20260925 13:21:41.726950 23176 log.cc:824] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5: Log is configured to *not* fsync() on all Append() calls
I20260925 13:21:41.728190 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5: No bootstrap required, opened a new log
I20260925 13:21:41.729687 23176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } }
I20260925 13:21:41.729808 23176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3895258b1f42f7a4adcc0d3be8f4f5, State: Initialized, Role: FOLLOWER
I20260925 13:21:41.729893 23176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } }
I20260925 13:21:41.730185 23300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } } }
I20260925 13:21:41.730262 23176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 13:21:41.730316 23150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "729697013d004b259b010cd5c17ed091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5"
I20260925 13:21:41.730445 23150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 13:21:41.731317 23150 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 729697013d004b259b010cd5c17ed091 in term 1.
I20260925 13:21:41.731556 23149 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5 [term 1 FOLLOWER]: Refusing update from remote peer 729697013d004b259b010cd5c17ed091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 13:21:41.732496 23288 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 13:21:41.734544 23314 mvcc.cc:205] Tried to move back new op lower bound from 7333242887089934336 to 7333242886941069312. Current Snapshot: MvccSnapshot[applied={T|T < 7333242886952243200}]
I20260925 13:21:41.735443 23313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 729697013d004b259b010cd5c17ed091. Latest consensus state: current_term: 1 leader_uuid: "729697013d004b259b010cd5c17ed091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } } }
I20260925 13:21:41.735532 23313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:21:41.735570 23313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "729697013d004b259b010cd5c17ed091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } } }
I20260925 13:21:41.735605 23313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:21:41.735677 23290 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 13:21:41.731895 23300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 13:21:41.736714 23290 catalog_manager.cc:1504] Loading token signing keys...
W20260925 13:21:41.738317 23311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 13:21:41.738427 23311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 13:21:41.747169 23271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "729697013d004b259b010cd5c17ed091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "729697013d004b259b010cd5c17ed091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42793 } } peers { permanent_uuid: "6e3895258b1f42f7a4adcc0d3be8f4f5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45039 } } peers { permanent_uuid: "bfbd89659e614029aec03ae01c529610" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44871 } } }
I20260925 13:21:41.747283 23271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091 [sys.catalog]: This master's current role is: LEADER
I20260925 13:21:41.749619 23290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 729697013d004b259b010cd5c17ed091: Generated new TSK 0
I20260925 13:21:41.749840 23290 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 13:21:41.774479 23180 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:41.772697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40664 (local address 127.0.98.61:45039)
0925 13:21:41.772747 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.772750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.773736 (+ 986us) server_negotiation.cc:407] Connection header received
0925 13:21:41.773850 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.773851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.773863 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.773894 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.774384 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40664: BlockingRecv error: recv got EOF from 127.0.0.1:40664 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 13:21:41.775292 23255 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:41.773084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.0.98.60:44871)
0925 13:21:41.773124 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.773126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.774724 (+ 1598us) server_negotiation.cc:407] Connection header received
0925 13:21:41.774819 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.774820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.774830 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.774858 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.775231 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44748: BlockingRecv error: recv got EOF from 127.0.0.1:44748 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 13:21:41.776063 23105 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:41.773444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.0.98.62:42793)
0925 13:21:41.773494 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.773495 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.775522 (+ 2027us) server_negotiation.cc:407] Connection header received
0925 13:21:41.775620 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.775621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.775632 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.775661 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.775969 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46060: BlockingRecv error: recv got EOF from 127.0.0.1:46060 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 13:21:41.778551 23180 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:41.777720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40674 (local address 127.0.98.61:45039)
0925 13:21:41.777758 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.777759 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.777988 (+ 229us) server_negotiation.cc:407] Connection header received
0925 13:21:41.778115 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.778116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.778136 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.778165 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.778498 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40674: BlockingRecv error: recv got EOF from 127.0.0.1:40674 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 13:21:41.781729 23180 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:41.780885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40690 (local address 127.0.98.61:45039)
0925 13:21:41.780918 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.780919 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.781161 (+ 242us) server_negotiation.cc:407] Connection header received
0925 13:21:41.781252 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.781253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.781264 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.781291 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.781680 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40690: BlockingRecv error: recv got EOF from 127.0.0.1:40690 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 13:21:41.785652 23180 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:41.784835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40702 (local address 127.0.98.61:45039)
0925 13:21:41.784870 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.784871 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.785117 (+ 246us) server_negotiation.cc:407] Connection header received
0925 13:21:41.785209 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.785210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.785222 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.785249 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.785586 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40702: BlockingRecv error: recv got EOF from 127.0.0.1:40702 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 13:21:41.791844 23180 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:41.790924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.0.98.61:45039)
0925 13:21:41.790968 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.790970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.791257 (+ 287us) server_negotiation.cc:407] Connection header received
0925 13:21:41.791362 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.791363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.791374 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.791404 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.791787 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40708: BlockingRecv error: recv got EOF from 127.0.0.1:40708 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 13:21:41.802153 23180 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:41.801169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40720 (local address 127.0.98.61:45039)
0925 13:21:41.801212 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.801214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.801508 (+ 294us) server_negotiation.cc:407] Connection header received
0925 13:21:41.801655 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.801656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.801669 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.801698 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.802090 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40720: BlockingRecv error: recv got EOF from 127.0.0.1:40720 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 13:21:41.820355 23180 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:41.819461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40736 (local address 127.0.98.61:45039)
0925 13:21:41.819505 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.819507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.819775 (+ 268us) server_negotiation.cc:407] Connection header received
0925 13:21:41.819872 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.819873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.819886 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.819914 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.820296 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40736: BlockingRecv error: recv got EOF from 127.0.0.1:40736 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 13:21:41.854288 23180 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:41.853628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.0.98.61:45039)
0925 13:21:41.853673 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.853675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.853809 (+ 134us) server_negotiation.cc:407] Connection header received
0925 13:21:41.853862 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.853863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.853875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.853929 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.854226 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40750: BlockingRecv error: recv got EOF from 127.0.0.1:40750 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 13:21:41.920058 23180 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:41.919391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.0.98.61:45039)
0925 13:21:41.919448 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:41.919450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:41.919537 (+ 87us) server_negotiation.cc:407] Connection header received
0925 13:21:41.919589 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:41.919590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:41.919601 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:41.919681 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:41.919996 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40756: BlockingRecv error: recv got EOF from 127.0.0.1:40756 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260925 13:21:42.049889 23180 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:42.049204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.0.98.61:45039)
0925 13:21:42.049276 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:42.049278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:42.049325 (+ 47us) server_negotiation.cc:407] Connection header received
0925 13:21:42.049371 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:42.049372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:42.049381 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:42.049450 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:42.049822 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40758: BlockingRecv error: recv got EOF from 127.0.0.1:40758 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 13:21:42.307627 23180 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 13:21:42.307076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.0.98.61:45039)
0925 13:21:42.307143 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:42.307145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:42.307154 (+ 9us) server_negotiation.cc:407] Connection header received
0925 13:21:42.307179 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:42.307180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:42.307189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:42.307240 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 13:21:42.307566 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40760: BlockingRecv error: recv got EOF from 127.0.0.1:40760 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 13:21:42.663703 23268 catalog_manager.cc:1259] Loaded cluster ID: bdeb46d1ee434e159007c5960eaee27a
I20260925 13:21:42.663800 23268 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610: loading cluster ID for follower catalog manager: success
I20260925 13:21:42.664636 23268 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610: acquiring CA information for follower catalog manager: success
I20260925 13:21:42.664925 23268 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bfbd89659e614029aec03ae01c529610: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 13:21:42.738893 23311 catalog_manager.cc:1259] Loaded cluster ID: bdeb46d1ee434e159007c5960eaee27a
I20260925 13:21:42.738966 23311 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5: loading cluster ID for follower catalog manager: success
I20260925 13:21:42.739727 23311 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5: acquiring CA information for follower catalog manager: success
I20260925 13:21:42.739984 23311 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6e3895258b1f42f7a4adcc0d3be8f4f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 13:21:42.823186 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:42.820684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.0.98.61:45039)
0925 13:21:42.820816 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:42.820818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:42.820829 (+ 11us) server_negotiation.cc:407] Connection header received
0925 13:21:42.820879 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:42.820880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:42.820929 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:42.820998 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:42.821144 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:42.821683 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:42.822442 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:42.822649 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:42.822704 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:42.822861 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:42.822977 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:42.823046 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:42.823070 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 13:21:42.823094 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
I20260925 13:21:42.828727 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:42.826199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44760 (local address 127.0.98.60:44871)
0925 13:21:42.826292 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:42.826294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:42.826301 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:21:42.826338 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:42.826339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:42.826373 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:42.826442 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:42.826611 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:42.827135 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:42.827914 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:42.828134 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:42.828187 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:42.828346 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:42.828450 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:42.828506 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:42.828634 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 13:21:42.828652 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":50,"threads_started":1}
I20260925 13:21:42.833216 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:42.830744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.0.98.60:44871)
0925 13:21:42.830784 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:42.830786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:42.830793 (+ 7us) server_negotiation.cc:407] Connection header received
0925 13:21:42.830811 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:42.830812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:42.830820 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:42.830877 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:42.831048 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:42.831635 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:42.832436 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:42.832687 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:42.832750 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:42.832867 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:42.832898 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:42.832943 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:42.833115 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 13:21:42.833136 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 13:21:42.833307 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:42.830461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.0.98.61:45039)
0925 13:21:42.830514 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:42.830515 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:42.830610 (+ 95us) server_negotiation.cc:407] Connection header received
0925 13:21:42.830699 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:42.830700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:42.830710 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:42.830737 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:42.830908 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:42.831591 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:42.832634 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:42.832901 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:42.832964 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:42.833055 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:42.833089 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:42.833128 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:42.833214 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 13:21:42.833237 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 13:21:42.834227 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 13:21:42.830438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.0.98.62:42793)
0925 13:21:42.830545 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 13:21:42.830547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 13:21:42.830604 (+ 57us) server_negotiation.cc:407] Connection header received
0925 13:21:42.830689 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 13:21:42.830690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 13:21:42.830742 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 13:21:42.830979 (+ 237us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 13:21:42.831731 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:42.832487 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:42.833371 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 13:21:42.833766 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 13:21:42.833831 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 13:21:42.833972 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 13:21:42.834058 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 13:21:42.834097 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 13:21:42.834126 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 13:21:42.834148 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":58,"threads_started":1}
I20260925 13:21:42.835659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 23033
I20260925 13:21:42.840214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 23108
I20260925 13:21:42.844982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bpD2/build/release/bin/kudu with pid 23183
2026-09-25T13:21:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2694 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5430 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (263015 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST